public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v1] Fix usage of unified logging pg_log_* in pg_rewind and initdb 124+ messages / 4 participants [nested] [flat]
* [PATCH v1] Fix usage of unified logging pg_log_* in pg_rewind and initdb @ 2019-07-01 15:11 Alexey Kondratov <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Alexey Kondratov @ 2019-07-01 15:11 UTC (permalink / raw) --- src/bin/initdb/initdb.c | 2 +- src/bin/pg_rewind/pg_rewind.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c index 2ef179165b..70273be783 100644 --- a/src/bin/initdb/initdb.c +++ b/src/bin/initdb/initdb.c @@ -2497,7 +2497,7 @@ setup_bin_paths(const char *argv0) pg_log_error("The program \"postgres\" is needed by %s but was not found in the\n" "same directory as \"%s\".\n" "Check your installation.", - full_path, progname); + progname, full_path); else pg_log_error("The program \"postgres\" was found by \"%s\"\n" "but was not the same version as %s.\n" diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index 6e77201be6..d378053de4 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -555,7 +555,7 @@ getTimelineHistory(ControlFileData *controlFile, int *nentries) else if (controlFile == &ControlFile_target) histfile = slurpFile(datadir_target, path, NULL); else - pg_fatal("invalid control file\n"); + pg_fatal("invalid control file"); history = rewind_parseTimeLineHistory(histfile, tli, nentries); pg_free(histfile); base-commit: 95bbe5d82e428db342fa3ec60b95f1b9873741e5 -- 2.17.1 --------------0DB63CE8F653DAD250E3D5A4-- ^ permalink raw reply [nested|flat] 124+ messages in thread
* Re: Schema variables - new implementation for Postgres 15 @ 2023-05-16 18:11 Pavel Stehule <[email protected]> 0 siblings, 1 reply; 124+ messages in thread From: Pavel Stehule @ 2023-05-16 18:11 UTC (permalink / raw) To: Dmitry Dolgov <[email protected]>; +Cc: Sergey Shinderuk <[email protected]>; Tomas Vondra <[email protected]>; Julien Rouhaud <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected] Hi still in old layout - but it can be useful for testing by someone fix build doc, fix regress tests Regards Pavel Attachments: [text/x-patch] v20230516-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch (27.2K, ../../CAFj8pRDF_eygHWcG8Y9d73zmus=ohVup-ngP7w-m4tbtWg14KQ@mail.gmail.com/3-v20230516-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch) download | inline diff: From aae46fc027969ca9752be95fe31b30d81c7bf9ae Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Thu, 24 Nov 2022 12:35:57 +0100 Subject: [PATCH 09/10] this patch changes error message "column doesn't exist" to message "column or variable doesn't exist" The error message will be more correct. Today, missing PL/pgSQL variable can be reported. The change has impact on lot of regress tests not related to session variables, and then it is distributed as separate patch --- src/backend/parser/parse_expr.c | 2 +- src/backend/parser/parse_relation.c | 24 +++++++++++--- src/backend/parser/parse_target.c | 8 +++-- src/include/parser/parse_expr.h | 1 + src/pl/plpgsql/src/expected/plpgsql_array.out | 2 +- .../plpgsql/src/expected/plpgsql_record.out | 4 +-- src/pl/tcl/expected/pltcl_queries.out | 12 +++---- src/test/regress/expected/alter_table.out | 32 +++++++++---------- src/test/regress/expected/copy2.out | 2 +- src/test/regress/expected/errors.out | 8 ++--- src/test/regress/expected/join.out | 12 +++---- src/test/regress/expected/namespace.out | 2 +- src/test/regress/expected/numerology.out | 2 +- src/test/regress/expected/plpgsql.out | 12 +++---- src/test/regress/expected/psql.out | 2 +- src/test/regress/expected/rules.out | 2 +- .../regress/expected/session_variables.out | 2 +- src/test/regress/expected/transactions.out | 4 +-- src/test/regress/expected/union.out | 2 +- 19 files changed, 77 insertions(+), 58 deletions(-) diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 7b7c2a16e5..608e7c191e 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -479,7 +479,7 @@ transformIndirection(ParseState *pstate, A_Indirection *ind) * When we are in an expression where session variables cannot be used, we raise * the first form of error message. */ -static bool +bool expr_kind_allows_session_variables(ParseExprKind p_expr_kind) { bool result = false; diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c index 41d60494b9..8f3399a42e 100644 --- a/src/backend/parser/parse_relation.c +++ b/src/backend/parser/parse_relation.c @@ -27,6 +27,7 @@ #include "nodes/makefuncs.h" #include "nodes/nodeFuncs.h" #include "parser/parse_enr.h" +#include "parser/parse_expr.h" #include "parser/parse_relation.h" #include "parser/parse_type.h" #include "parser/parsetree.h" @@ -3651,6 +3652,19 @@ errorMissingRTE(ParseState *pstate, RangeVar *relation) parser_errposition(pstate, relation->location))); } +/* + * set message "column does not exist" or "column or variable does not exist" + * in dependency if expression context allows session variables. + */ +static int +column_or_variable_does_not_exists(ParseState *pstate, const char *colname) +{ + if (expr_kind_allows_session_variables(pstate->p_expr_kind)) + return errmsg("column or variable \"%s\" does not exist", colname); + else + return errmsg("column \"%s\" does not exist", colname); +} + /* * Generate a suitable error about a missing column. * @@ -3685,7 +3699,7 @@ errorMissingColumn(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), relname ? errmsg("column %s.%s does not exist", relname, colname) : - errmsg("column \"%s\" does not exist", colname), + column_or_variable_does_not_exists(pstate, colname), errdetail("There are columns named \"%s\", but they are in tables that cannot be referenced from this part of the query.", colname), !relname ? errhint("Try using a table-qualified name.") : 0, @@ -3695,7 +3709,7 @@ errorMissingColumn(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), relname ? errmsg("column %s.%s does not exist", relname, colname) : - errmsg("column \"%s\" does not exist", colname), + column_or_variable_does_not_exists(pstate, colname), errdetail("There is a column named \"%s\" in table \"%s\", but it cannot be referenced from this part of the query.", colname, state->rexact1->eref->aliasname), rte_visible_if_lateral(pstate, state->rexact1) ? @@ -3713,14 +3727,14 @@ errorMissingColumn(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), relname ? errmsg("column %s.%s does not exist", relname, colname) : - errmsg("column \"%s\" does not exist", colname), + column_or_variable_does_not_exists(pstate, colname), parser_errposition(pstate, location))); /* Handle case where we have a single alternative spelling to offer */ ereport(ERROR, (errcode(ERRCODE_UNDEFINED_COLUMN), relname ? errmsg("column %s.%s does not exist", relname, colname) : - errmsg("column \"%s\" does not exist", colname), + column_or_variable_does_not_exists(pstate, colname), errhint("Perhaps you meant to reference the column \"%s.%s\".", state->rfirst->eref->aliasname, strVal(list_nth(state->rfirst->eref->colnames, @@ -3734,7 +3748,7 @@ errorMissingColumn(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), relname ? errmsg("column %s.%s does not exist", relname, colname) : - errmsg("column \"%s\" does not exist", colname), + column_or_variable_does_not_exists(pstate, colname), errhint("Perhaps you meant to reference the column \"%s.%s\" or the column \"%s.%s\".", state->rfirst->eref->aliasname, strVal(list_nth(state->rfirst->eref->colnames, diff --git a/src/backend/parser/parse_target.c b/src/backend/parser/parse_target.c index ef5601ef4c..25b8935445 100644 --- a/src/backend/parser/parse_target.c +++ b/src/backend/parser/parse_target.c @@ -779,7 +779,9 @@ transformAssignmentIndirection(ParseState *pstate, if (!typrelid) ereport(ERROR, (errcode(ERRCODE_DATATYPE_MISMATCH), - errmsg("cannot assign to field \"%s\" of column \"%s\" because its type %s is not a composite type", + errmsg(expr_kind_allows_session_variables(pstate->p_expr_kind) ? + "cannot assign to field \"%s\" of column or variable \"%s\" because its type %s is not a composite type" : + "cannot assign to field \"%s\" of column \"%s\" because its type %s is not a composite type", strVal(n), targetName, format_type_be(targetTypeId)), parser_errposition(pstate, location))); @@ -788,7 +790,9 @@ transformAssignmentIndirection(ParseState *pstate, if (attnum == InvalidAttrNumber) ereport(ERROR, (errcode(ERRCODE_UNDEFINED_COLUMN), - errmsg("cannot assign to field \"%s\" of column \"%s\" because there is no such column in data type %s", + errmsg(expr_kind_allows_session_variables(pstate->p_expr_kind) ? + "cannot assign to field \"%s\" of column or variable \"%s\" because there is no such column in data type %s" : + "cannot assign to field \"%s\" of column \"%s\" because there is no such column in data type %s", strVal(n), targetName, format_type_be(targetTypeId)), parser_errposition(pstate, location))); diff --git a/src/include/parser/parse_expr.h b/src/include/parser/parse_expr.h index f6a5ccb0c1..ebd9ed7cbb 100644 --- a/src/include/parser/parse_expr.h +++ b/src/include/parser/parse_expr.h @@ -22,5 +22,6 @@ extern PGDLLIMPORT bool session_variables_ambiguity_warning; extern Node *transformExpr(ParseState *pstate, Node *expr, ParseExprKind exprKind); extern const char *ParseExprKindName(ParseExprKind exprKind); +extern bool expr_kind_allows_session_variables(ParseExprKind p_expr_kind); #endif /* PARSE_EXPR_H */ diff --git a/src/pl/plpgsql/src/expected/plpgsql_array.out b/src/pl/plpgsql/src/expected/plpgsql_array.out index 9e22e56f00..e131febf3d 100644 --- a/src/pl/plpgsql/src/expected/plpgsql_array.out +++ b/src/pl/plpgsql/src/expected/plpgsql_array.out @@ -41,7 +41,7 @@ NOTICE: a = {"(,11)"}, a[1].i = 11 -- perhaps this ought to work, but for now it doesn't: do $$ declare a complex[]; begin a[1:2].i := array[11,12]; raise notice 'a = %', a; end$$; -ERROR: cannot assign to field "i" of column "a" because its type complex[] is not a composite type +ERROR: cannot assign to field "i" of column or variable "a" because its type complex[] is not a composite type LINE 1: a[1:2].i := array[11,12] ^ QUERY: a[1:2].i := array[11,12] diff --git a/src/pl/plpgsql/src/expected/plpgsql_record.out b/src/pl/plpgsql/src/expected/plpgsql_record.out index afb922df29..f702c7ad54 100644 --- a/src/pl/plpgsql/src/expected/plpgsql_record.out +++ b/src/pl/plpgsql/src/expected/plpgsql_record.out @@ -135,7 +135,7 @@ ERROR: record "c" has no field "x" CONTEXT: PL/pgSQL assignment "c.x.q1 = 1" PL/pgSQL function inline_code_block line 1 at assignment do $$ declare c nested_int8s; begin c.c2.x = 1; end $$; -ERROR: cannot assign to field "x" of column "c" because there is no such column in data type two_int8s +ERROR: cannot assign to field "x" of column or variable "c" because there is no such column in data type two_int8s LINE 1: c.c2.x = 1 ^ QUERY: c.c2.x = 1 @@ -157,7 +157,7 @@ ERROR: record "c" has no field "x" CONTEXT: PL/pgSQL assignment "b.c.x.q1 = 1" PL/pgSQL function inline_code_block line 1 at assignment do $$ <<b>> declare c nested_int8s; begin b.c.c2.x = 1; end $$; -ERROR: cannot assign to field "x" of column "b" because there is no such column in data type two_int8s +ERROR: cannot assign to field "x" of column or variable "b" because there is no such column in data type two_int8s LINE 1: b.c.c2.x = 1 ^ QUERY: b.c.c2.x = 1 diff --git a/src/pl/tcl/expected/pltcl_queries.out b/src/pl/tcl/expected/pltcl_queries.out index 2d922c2333..3b6506d613 100644 --- a/src/pl/tcl/expected/pltcl_queries.out +++ b/src/pl/tcl/expected/pltcl_queries.out @@ -246,12 +246,12 @@ ERROR: type "b" does not exist select tcl_eval('spi_prepare a "b {"'); ERROR: unmatched open brace in list select tcl_error_handling_test($tcl$spi_prepare "select moo" []$tcl$); - tcl_error_handling_test --------------------------------------- - SQLSTATE: 42703 + - condition: undefined_column + - cursor_position: 8 + - message: column "moo" does not exist+ + tcl_error_handling_test +-------------------------------------------------- + SQLSTATE: 42703 + + condition: undefined_column + + cursor_position: 8 + + message: column or variable "moo" does not exist+ statement: select moo (1 row) diff --git a/src/test/regress/expected/alter_table.out b/src/test/regress/expected/alter_table.out index 3b708c7976..def3039c44 100644 --- a/src/test/regress/expected/alter_table.out +++ b/src/test/regress/expected/alter_table.out @@ -1287,19 +1287,19 @@ select * from atacc1; (1 row) select * from atacc1 order by a; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: select * from atacc1 order by a; ^ select * from atacc1 order by "........pg.dropped.1........"; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: select * from atacc1 order by "........pg.dropped.1........"... ^ select * from atacc1 group by a; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: select * from atacc1 group by a; ^ select * from atacc1 group by "........pg.dropped.1........"; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: select * from atacc1 group by "........pg.dropped.1........"... ^ select atacc1.* from atacc1; @@ -1309,7 +1309,7 @@ select atacc1.* from atacc1; (1 row) select a from atacc1; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: select a from atacc1; ^ select atacc1.a from atacc1; @@ -1323,15 +1323,15 @@ select b,c,d from atacc1; (1 row) select a,b,c,d from atacc1; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: select a,b,c,d from atacc1; ^ select * from atacc1 where a = 1; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: select * from atacc1 where a = 1; ^ select "........pg.dropped.1........" from atacc1; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: select "........pg.dropped.1........" from atacc1; ^ select atacc1."........pg.dropped.1........" from atacc1; @@ -1339,11 +1339,11 @@ ERROR: column atacc1.........pg.dropped.1........ does not exist LINE 1: select atacc1."........pg.dropped.1........" from atacc1; ^ select "........pg.dropped.1........",b,c,d from atacc1; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: select "........pg.dropped.1........",b,c,d from atacc1; ^ select * from atacc1 where "........pg.dropped.1........" = 1; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: select * from atacc1 where "........pg.dropped.1........" = ... ^ -- UPDATEs @@ -1352,7 +1352,7 @@ ERROR: column "a" of relation "atacc1" does not exist LINE 1: update atacc1 set a = 3; ^ update atacc1 set b = 2 where a = 3; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: update atacc1 set b = 2 where a = 3; ^ update atacc1 set "........pg.dropped.1........" = 3; @@ -1360,7 +1360,7 @@ ERROR: column "........pg.dropped.1........" of relation "atacc1" does not exis LINE 1: update atacc1 set "........pg.dropped.1........" = 3; ^ update atacc1 set b = 2 where "........pg.dropped.1........" = 3; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: update atacc1 set b = 2 where "........pg.dropped.1........"... ^ -- INSERTs @@ -1408,11 +1408,11 @@ LINE 1: insert into atacc1 ("........pg.dropped.1........",b,c,d) va... ^ -- DELETEs delete from atacc1 where a = 3; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: delete from atacc1 where a = 3; ^ delete from atacc1 where "........pg.dropped.1........" = 3; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: delete from atacc1 where "........pg.dropped.1........" = 3; ^ delete from atacc1; @@ -1698,7 +1698,7 @@ select f1 from c1; alter table c1 drop column f1; select f1 from c1; -ERROR: column "f1" does not exist +ERROR: column or variable "f1" does not exist LINE 1: select f1 from c1; ^ HINT: Perhaps you meant to reference the column "c1.f2". @@ -1712,7 +1712,7 @@ ERROR: cannot drop inherited column "f1" alter table p1 drop column f1; -- c1.f1 is dropped now, since there is no local definition for it select f1 from c1; -ERROR: column "f1" does not exist +ERROR: column or variable "f1" does not exist LINE 1: select f1 from c1; ^ HINT: Perhaps you meant to reference the column "c1.f2". diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out index 8e33eee719..9b11d87fc6 100644 --- a/src/test/regress/expected/copy2.out +++ b/src/test/regress/expected/copy2.out @@ -122,7 +122,7 @@ LINE 1: COPY x TO stdout WHERE a = 1; COPY x from stdin WHERE a = 50004; COPY x from stdin WHERE a > 60003; COPY x from stdin WHERE f > 60003; -ERROR: column "f" does not exist +ERROR: column or variable "f" does not exist LINE 1: COPY x from stdin WHERE f > 60003; ^ COPY x from stdin WHERE a = max(x.b); diff --git a/src/test/regress/expected/errors.out b/src/test/regress/expected/errors.out index 8c527474da..e53ae451df 100644 --- a/src/test/regress/expected/errors.out +++ b/src/test/regress/expected/errors.out @@ -27,7 +27,7 @@ LINE 1: select * from nonesuch; ^ -- bad name in target list select nonesuch from pg_database; -ERROR: column "nonesuch" does not exist +ERROR: column or variable "nonesuch" does not exist LINE 1: select nonesuch from pg_database; ^ -- empty distinct list isn't OK @@ -37,17 +37,17 @@ LINE 1: select distinct from pg_database; ^ -- bad attribute name on lhs of operator select * from pg_database where nonesuch = pg_database.datname; -ERROR: column "nonesuch" does not exist +ERROR: column or variable "nonesuch" does not exist LINE 1: select * from pg_database where nonesuch = pg_database.datna... ^ -- bad attribute name on rhs of operator select * from pg_database where pg_database.datname = nonesuch; -ERROR: column "nonesuch" does not exist +ERROR: column or variable "nonesuch" does not exist LINE 1: ...ect * from pg_database where pg_database.datname = nonesuch; ^ -- bad attribute name in select distinct on select distinct on (foobar) * from pg_database; -ERROR: column "foobar" does not exist +ERROR: column or variable "foobar" does not exist LINE 1: select distinct on (foobar) * from pg_database; ^ -- grouping with FOR UPDATE diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out index b5f440e43e..51d145edf0 100644 --- a/src/test/regress/expected/join.out +++ b/src/test/regress/expected/join.out @@ -5678,13 +5678,13 @@ LINE 1: select t2.uunique1 from HINT: Perhaps you meant to reference the column "t2.unique1". select uunique1 from tenk1 t1 join tenk2 t2 on t1.two = t2.two; -- error, suggest both at once -ERROR: column "uunique1" does not exist +ERROR: column or variable "uunique1" does not exist LINE 1: select uunique1 from ^ HINT: Perhaps you meant to reference the column "t1.unique1" or the column "t2.unique1". select ctid from tenk1 t1 join tenk2 t2 on t1.two = t2.two; -- error, need qualification -ERROR: column "ctid" does not exist +ERROR: column or variable "ctid" does not exist LINE 1: select ctid from ^ DETAIL: There are columns named "ctid", but they are in tables that cannot be referenced from this part of the query. @@ -6754,7 +6754,7 @@ lateral (select * from int8_tbl t1, -- test some error cases where LATERAL should have been used but wasn't select f1,g from int4_tbl a, (select f1 as g) ss; -ERROR: column "f1" does not exist +ERROR: column or variable "f1" does not exist LINE 1: select f1,g from int4_tbl a, (select f1 as g) ss; ^ DETAIL: There is a column named "f1" in table "a", but it cannot be referenced from this part of the query. @@ -6766,7 +6766,7 @@ LINE 1: select f1,g from int4_tbl a, (select a.f1 as g) ss; DETAIL: There is an entry for table "a", but it cannot be referenced from this part of the query. HINT: To reference that table, you must mark this subquery with LATERAL. select f1,g from int4_tbl a cross join (select f1 as g) ss; -ERROR: column "f1" does not exist +ERROR: column or variable "f1" does not exist LINE 1: select f1,g from int4_tbl a cross join (select f1 as g) ss; ^ DETAIL: There is a column named "f1" in table "a", but it cannot be referenced from this part of the query. @@ -6803,7 +6803,7 @@ LINE 1: select 1 from tenk1 a, lateral (select max(a.unique1) from i... create temp table xx1 as select f1 as x1, -f1 as x2 from int4_tbl; -- error, can't do this: update xx1 set x2 = f1 from (select * from int4_tbl where f1 = x1) ss; -ERROR: column "x1" does not exist +ERROR: column or variable "x1" does not exist LINE 1: ... set x2 = f1 from (select * from int4_tbl where f1 = x1) ss; ^ DETAIL: There is a column named "x1" in table "xx1", but it cannot be referenced from this part of the query. @@ -6823,7 +6823,7 @@ update xx1 set x2 = f1 from xx1, lateral (select * from int4_tbl where f1 = x1) ERROR: table name "xx1" specified more than once -- also errors: delete from xx1 using (select * from int4_tbl where f1 = x1) ss; -ERROR: column "x1" does not exist +ERROR: column or variable "x1" does not exist LINE 1: ...te from xx1 using (select * from int4_tbl where f1 = x1) ss; ^ DETAIL: There is a column named "x1" in table "xx1", but it cannot be referenced from this part of the query. diff --git a/src/test/regress/expected/namespace.out b/src/test/regress/expected/namespace.out index a62fd8ded0..1966b55fea 100644 --- a/src/test/regress/expected/namespace.out +++ b/src/test/regress/expected/namespace.out @@ -23,7 +23,7 @@ BEGIN; SET search_path to public, test_ns_schema_1; CREATE SCHEMA test_ns_schema_2 CREATE VIEW abc_view AS SELECT c FROM abc; -ERROR: column "c" does not exist +ERROR: column or variable "c" does not exist LINE 2: CREATE VIEW abc_view AS SELECT c FROM abc; ^ COMMIT; diff --git a/src/test/regress/expected/numerology.out b/src/test/regress/expected/numerology.out index f662a5050a..fd89def959 100644 --- a/src/test/regress/expected/numerology.out +++ b/src/test/regress/expected/numerology.out @@ -299,7 +299,7 @@ SELECT 1_000.5e0_1; -- error cases SELECT _100; -ERROR: column "_100" does not exist +ERROR: column or variable "_100" does not exist LINE 1: SELECT _100; ^ SELECT 100_; diff --git a/src/test/regress/expected/plpgsql.out b/src/test/regress/expected/plpgsql.out index 272f5d2111..44c3a35461 100644 --- a/src/test/regress/expected/plpgsql.out +++ b/src/test/regress/expected/plpgsql.out @@ -2600,7 +2600,7 @@ end; $$ language plpgsql; -- should fail: SQLSTATE and SQLERRM are only in defined EXCEPTION -- blocks select excpt_test1(); -ERROR: column "sqlstate" does not exist +ERROR: column or variable "sqlstate" does not exist LINE 1: sqlstate ^ QUERY: sqlstate @@ -2615,7 +2615,7 @@ begin end; $$ language plpgsql; -- should fail select excpt_test2(); -ERROR: column "sqlstate" does not exist +ERROR: column or variable "sqlstate" does not exist LINE 1: sqlstate ^ QUERY: sqlstate @@ -4648,7 +4648,7 @@ BEGIN RAISE NOTICE '%, %', r.roomno, r.comment; END LOOP; END$$; -ERROR: column "foo" does not exist +ERROR: column or variable "foo" does not exist LINE 1: SELECT rtrim(roomno) AS roomno, foo FROM Room ORDER BY roomn... ^ QUERY: SELECT rtrim(roomno) AS roomno, foo FROM Room ORDER BY roomno @@ -4690,7 +4690,7 @@ begin raise notice 'x = %', x; end; $$; -ERROR: column "x" does not exist +ERROR: column or variable "x" does not exist LINE 1: x + 1 ^ QUERY: x + 1 @@ -4702,7 +4702,7 @@ begin raise notice 'x = %, y = %', x, y; end; $$; -ERROR: column "x" does not exist +ERROR: column or variable "x" does not exist LINE 1: x + 1 ^ QUERY: x + 1 @@ -5742,7 +5742,7 @@ ALTER TABLE alter_table_under_transition_tables DROP column name; UPDATE alter_table_under_transition_tables SET id = id; -ERROR: column "name" does not exist +ERROR: column or variable "name" does not exist LINE 1: (SELECT string_agg(id || '=' || name, ',') FROM d) ^ QUERY: (SELECT string_agg(id || '=' || name, ',') FROM d) diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 956e475447..1fd706a6d0 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -120,7 +120,7 @@ SELECT $1, $2 \bind 'foo' 'bar' \g -- errors -- parse error SELECT foo \bind \g -ERROR: column "foo" does not exist +ERROR: column or variable "foo" does not exist LINE 1: SELECT foo ^ -- tcop error diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 764b827372..22d82333b4 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1191,7 +1191,7 @@ drop rule rules_foorule on rules_foo; -- this should fail because f1 is not exposed for unqualified reference: create rule rules_foorule as on insert to rules_foo where f1 < 100 do instead insert into rules_foo2 values (f1); -ERROR: column "f1" does not exist +ERROR: column or variable "f1" does not exist LINE 2: do instead insert into rules_foo2 values (f1); ^ DETAIL: There are columns named "f1", but they are in tables that cannot be referenced from this part of the query. diff --git a/src/test/regress/expected/session_variables.out b/src/test/regress/expected/session_variables.out index 463b2c96f6..01c234c6f4 100644 --- a/src/test/regress/expected/session_variables.out +++ b/src/test/regress/expected/session_variables.out @@ -273,7 +273,7 @@ SELECT v1; -- should fail LET v1.x = 10; -ERROR: cannot assign to field "x" of column "v1" because there is no such column in data type t1 +ERROR: cannot assign to field "x" of column or variable "v1" because there is no such column in data type t1 LINE 1: LET v1.x = 10; ^ DROP VARIABLE v1; diff --git a/src/test/regress/expected/transactions.out b/src/test/regress/expected/transactions.out index 2b2cff7d91..60c096b857 100644 --- a/src/test/regress/expected/transactions.out +++ b/src/test/regress/expected/transactions.out @@ -256,7 +256,7 @@ SELECT * FROM trans_barbaz; -- should have 1 BEGIN; SAVEPOINT one; SELECT trans_foo; -ERROR: column "trans_foo" does not exist +ERROR: column or variable "trans_foo" does not exist LINE 1: SELECT trans_foo; ^ ROLLBACK TO SAVEPOINT one; @@ -305,7 +305,7 @@ BEGIN; SAVEPOINT one; INSERT INTO savepoints VALUES (5); SELECT trans_foo; -ERROR: column "trans_foo" does not exist +ERROR: column or variable "trans_foo" does not exist LINE 1: SELECT trans_foo; ^ COMMIT; diff --git a/src/test/regress/expected/union.out b/src/test/regress/expected/union.out index e2613d6777..0a6613c430 100644 --- a/src/test/regress/expected/union.out +++ b/src/test/regress/expected/union.out @@ -907,7 +907,7 @@ ORDER BY q2,q1; -- This should fail, because q2 isn't a name of an EXCEPT output column SELECT q1 FROM int8_tbl EXCEPT SELECT q2 FROM int8_tbl ORDER BY q2 LIMIT 1; -ERROR: column "q2" does not exist +ERROR: column or variable "q2" does not exist LINE 1: ... int8_tbl EXCEPT SELECT q2 FROM int8_tbl ORDER BY q2 LIMIT 1... ^ DETAIL: There is a column named "q2" in table "*SELECT* 2", but it cannot be referenced from this part of the query. -- 2.40.1 [text/x-patch] v20230516-1-0006-enhancing-psql-for-session-variables.patch (14.1K, ../../CAFj8pRDF_eygHWcG8Y9d73zmus=ohVup-ngP7w-m4tbtWg14KQ@mail.gmail.com/4-v20230516-1-0006-enhancing-psql-for-session-variables.patch) download | inline diff: From 3fa38bd74809409ba44446fe87a5e062443cc4ed Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Mon, 4 Apr 2022 22:40:13 +0200 Subject: [PATCH 06/10] enhancing psql for session variables \dV and tab complete for session variables related commands --- src/bin/psql/command.c | 3 ++ src/bin/psql/describe.c | 96 +++++++++++++++++++++++++++++++++++++ src/bin/psql/describe.h | 3 ++ src/bin/psql/help.c | 1 + src/bin/psql/tab-complete.c | 64 +++++++++++++++++++++---- 5 files changed, 157 insertions(+), 10 deletions(-) diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c index 607a57715a..f53244ada3 100644 --- a/src/bin/psql/command.c +++ b/src/bin/psql/command.c @@ -981,6 +981,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd) break; } break; + case 'V': /* Variables */ + success = listVariables(pattern, show_verbose); + break; case 'x': /* Extensions */ if (show_verbose) success = listExtensionContents(pattern); diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 50f39a2e50..82a7e20f38 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -5089,6 +5089,102 @@ error_return: return false; } +/* + * \dV + * + * listVariables() + */ +bool +listVariables(const char *pattern, bool verbose) +{ + PQExpBufferData buf; + PGresult *res; + printQueryOpt myopt = pset.popt; + static const bool translate_columns[] = {false, false, false, false, false, false, false, false, false, false, false}; + + initPQExpBuffer(&buf); + + printfPQExpBuffer(&buf, + "SELECT n.nspname as \"%s\",\n" + " v.varname as \"%s\",\n" + " pg_catalog.format_type(v.vartype, v.vartypmod) as \"%s\",\n" + " (SELECT c.collname FROM pg_catalog.pg_collation c, pg_catalog.pg_type bt\n" + " WHERE c.oid = v.varcollation AND bt.oid = v.vartype AND v.varcollation <> bt.typcollation) as \"%s\",\n" + " NOT v.varisnotnull as \"%s\",\n" + " NOT v.varisimmutable as \"%s\",\n" + " pg_catalog.pg_get_expr(v.vardefexpr, 0) as \"%s\",\n" + " pg_catalog.pg_get_userbyid(v.varowner) as \"%s\",\n" + " CASE v.vareoxaction\n" + " WHEN 'd' THEN 'ON COMMIT DROP'\n" + " WHEN 'r' THEN 'ON TRANSACTION END RESET' END as \"%s\"\n", + gettext_noop("Schema"), + gettext_noop("Name"), + gettext_noop("Type"), + gettext_noop("Collation"), + gettext_noop("Nullable"), + gettext_noop("Mutable"), + gettext_noop("Default"), + gettext_noop("Owner"), + gettext_noop("Transactional end action")); + + if (verbose) + { + appendPQExpBufferStr(&buf, ",\n "); + printACLColumn(&buf, "v.varacl"); + appendPQExpBuffer(&buf, + ",\n pg_catalog.obj_description(v.oid, 'pg_variable') AS \"%s\"", + gettext_noop("Description")); + } + + appendPQExpBufferStr(&buf, + "\nFROM pg_catalog.pg_variable v" + "\n LEFT JOIN pg_catalog.pg_namespace n ON n.oid = v.varnamespace"); + + appendPQExpBufferStr(&buf, "\nWHERE true\n"); + if (!pattern) + appendPQExpBufferStr(&buf, " AND n.nspname <> 'pg_catalog'\n" + " AND n.nspname <> 'information_schema'\n"); + + if (!validateSQLNamePattern(&buf, pattern, true, false, + "n.nspname", "v.varname", NULL, + "pg_catalog.pg_variable_is_visible(v.oid)", + NULL, 3)) + return false; + + appendPQExpBufferStr(&buf, "ORDER BY 1,2;"); + + res = PSQLexec(buf.data); + termPQExpBuffer(&buf); + if (!res) + return false; + + /* + * Most functions in this file are content to print an empty table when + * there are no matching objects. We intentionally deviate from that + * here, but only in !quiet mode, for historical reasons. + */ + if (PQntuples(res) == 0 && !pset.quiet) + { + if (pattern) + pg_log_error("Did not find any session variable named \"%s\".", + pattern); + else + pg_log_error("Did not find any session variables."); + } + else + { + myopt.nullPrint = NULL; + myopt.title = _("List of variables"); + myopt.translate_header = true; + myopt.translate_columns = translate_columns; + myopt.n_translate_columns = lengthof(translate_columns); + + printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + } + + PQclear(res); + return true; +} /* * \dFp diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h index 554fe86725..55ae238ab2 100644 --- a/src/bin/psql/describe.h +++ b/src/bin/psql/describe.h @@ -146,4 +146,7 @@ extern bool listOpFamilyFunctions(const char *access_method_pattern, /* \dl or \lo_list */ extern bool listLargeObjects(bool verbose); +/* \dV */ +extern bool listVariables(const char *pattern, bool varbose); + #endif /* DESCRIBE_H */ diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index ecfb3c099b..7e2d65c811 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -287,6 +287,7 @@ slashUsage(unsigned short int pager) HELP0(" \\dT[S+] [PATTERN] list data types\n"); HELP0(" \\du[S+] [PATTERN] list roles\n"); HELP0(" \\dv[S+] [PATTERN] list views\n"); + HELP0(" \\dV [PATTERN] list variables\n"); HELP0(" \\dx[+] [PATTERN] list extensions\n"); HELP0(" \\dX [PATTERN] list extended statistics\n"); HELP0(" \\dy[+] [PATTERN] list event triggers\n"); diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index bd04244969..e7caa2ea72 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -962,6 +962,13 @@ static const SchemaQuery Query_for_trigger_of_table = { .refnamespace = "c1.relnamespace", }; +static const SchemaQuery Query_for_list_of_variables = { + .min_server_version = 150000, + .catname = "pg_catalog.pg_variable v", + .viscondition = "pg_catalog.pg_variable_is_visible(v.oid)", + .namespace = "v.varnamespace", + .result = "v.varname", +}; /* * Queries to get lists of names of various kinds of things, possibly @@ -1240,6 +1247,8 @@ static const pgsql_thing_t words_after_create[] = { {"FOREIGN TABLE", NULL, NULL, NULL}, {"FUNCTION", NULL, NULL, Query_for_list_of_functions}, {"GROUP", Query_for_list_of_roles}, + {"IMMUTABLE VARIABLE", NULL, NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE IMMUTABLE + * VARIABLE ... */ {"INDEX", NULL, NULL, &Query_for_list_of_indexes}, {"LANGUAGE", Query_for_list_of_languages}, {"LARGE OBJECT", NULL, NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP}, @@ -1278,6 +1287,7 @@ static const pgsql_thing_t words_after_create[] = { * TABLE ... */ {"USER", Query_for_list_of_roles, NULL, NULL, Keywords_for_user_thing}, {"USER MAPPING FOR", NULL, NULL, NULL}, + {"VARIABLE", NULL, NULL, &Query_for_list_of_variables}, {"VIEW", NULL, NULL, &Query_for_list_of_views}, {NULL} /* end of list */ }; @@ -1689,8 +1699,8 @@ psql_completion(const char *text, int start, int end) "ABORT", "ALTER", "ANALYZE", "BEGIN", "CALL", "CHECKPOINT", "CLOSE", "CLUSTER", "COMMENT", "COMMIT", "COPY", "CREATE", "DEALLOCATE", "DECLARE", "DELETE FROM", "DISCARD", "DO", "DROP", "END", "EXECUTE", "EXPLAIN", - "FETCH", "GRANT", "IMPORT FOREIGN SCHEMA", "INSERT INTO", "LISTEN", "LOAD", "LOCK", - "MERGE INTO", "MOVE", "NOTIFY", "PREPARE", + "FETCH", "GRANT", "IMPORT FOREIGN SCHEMA", "INSERT INTO", "LET", + "LISTEN", "LOAD", "LOCK", "MERGE INTO", "MOVE", "NOTIFY", "PREPARE", "REASSIGN", "REFRESH MATERIALIZED VIEW", "REINDEX", "RELEASE", "RESET", "REVOKE", "ROLLBACK", "SAVEPOINT", "SECURITY LABEL", "SELECT", "SET", "SHOW", "START", @@ -1710,7 +1720,7 @@ psql_completion(const char *text, int start, int end) "\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL", "\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt", "\\drds", "\\dRs", "\\dRp", "\\ds", - "\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy", + "\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy", "\\dV", "\\echo", "\\edit", "\\ef", "\\elif", "\\else", "\\encoding", "\\endif", "\\errverbose", "\\ev", "\\f", @@ -2203,6 +2213,9 @@ psql_completion(const char *text, int start, int end) "ALL"); else if (Matches("ALTER", "SYSTEM", "SET", MatchAny)) COMPLETE_WITH("TO"); + /* ALTER VARIABLE <name> */ + else if (Matches("ALTER", "VARIABLE", MatchAny)) + COMPLETE_WITH("OWNER TO", "RENAME TO", "SET SCHEMA"); /* ALTER VIEW <name> */ else if (Matches("ALTER", "VIEW", MatchAny)) COMPLETE_WITH("ALTER COLUMN", "OWNER TO", "RENAME", @@ -2745,7 +2758,7 @@ psql_completion(const char *text, int start, int end) "ROUTINE", "RULE", "SCHEMA", "SEQUENCE", "SERVER", "STATISTICS", "SUBSCRIPTION", "TABLE", "TABLESPACE", "TEXT SEARCH", "TRANSFORM FOR", - "TRIGGER", "TYPE", "VIEW"); + "TRIGGER", "TYPE", "VARIABLE", "VIEW"); else if (Matches("COMMENT", "ON", "ACCESS", "METHOD")) COMPLETE_WITH_QUERY(Query_for_list_of_access_methods); else if (Matches("COMMENT", "ON", "CONSTRAINT")) @@ -3184,7 +3197,7 @@ psql_completion(const char *text, int start, int end) /* CREATE TABLE --- is allowed inside CREATE SCHEMA, so use TailMatches */ /* Complete "CREATE TEMP/TEMPORARY" with the possible temp objects */ else if (TailMatches("CREATE", "TEMP|TEMPORARY")) - COMPLETE_WITH("SEQUENCE", "TABLE", "VIEW"); + COMPLETE_WITH("IMMUTABLE VARIABLE", "SEQUENCE", "TABLE", "VIEW", "VARIABLE"); /* Complete "CREATE UNLOGGED" with TABLE or MATVIEW */ else if (TailMatches("CREATE", "UNLOGGED")) COMPLETE_WITH("TABLE", "MATERIALIZED VIEW"); @@ -3492,6 +3505,17 @@ psql_completion(const char *text, int start, int end) else if (TailMatches("=", MatchAnyExcept("*)"))) COMPLETE_WITH(",", ")"); } +/* CREATE VARIABLE --- is allowed inside CREATE SCHEMA, so use TailMatches */ + /* Complete CREATE VARIABLE <name> with AS */ + else if (TailMatches("IMMUTABLE")) + COMPLETE_WITH("VARIABLE"); + else if (TailMatches("CREATE", "VARIABLE", MatchAny) || + TailMatches("TEMP|TEMPORARY", "VARIABLE", MatchAny) || + TailMatches("IMMUTABLE", "VARIABLE", MatchAny)) + COMPLETE_WITH("AS"); + else if (TailMatches("VARIABLE", MatchAny, "AS")) + /* Complete CREATE VARIABLE <name> with AS types */ + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_datatypes); /* CREATE VIEW --- is allowed inside CREATE SCHEMA, so use TailMatches */ /* Complete CREATE [ OR REPLACE ] VIEW <name> with AS */ @@ -3607,7 +3631,7 @@ psql_completion(const char *text, int start, int end) /* DISCARD */ else if (Matches("DISCARD")) - COMPLETE_WITH("ALL", "PLANS", "SEQUENCES", "TEMP"); + COMPLETE_WITH("ALL", "PLANS", "SEQUENCES", "TEMP", "VARIABLES"); /* DO */ else if (Matches("DO")) @@ -3734,6 +3758,12 @@ psql_completion(const char *text, int start, int end) else if (Matches("DROP", "TRANSFORM", "FOR", MatchAny, "LANGUAGE", MatchAny)) COMPLETE_WITH("CASCADE", "RESTRICT"); + /* DROP VARIABLE */ + else if (Matches("DROP", "VARIABLE")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_variables); + else if (Matches("DROP", "VARIABLE", MatchAny)) + COMPLETE_WITH("CASCADE", "RESTRICT"); + /* EXECUTE */ else if (Matches("EXECUTE")) COMPLETE_WITH_QUERY(Query_for_list_of_prepared_statements); @@ -3925,7 +3955,7 @@ psql_completion(const char *text, int start, int end) * objects supported. */ if (HeadMatches("ALTER", "DEFAULT", "PRIVILEGES")) - COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS"); + COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS", "VARIABLES"); else COMPLETE_WITH_SCHEMA_QUERY_PLUS(Query_for_list_of_grantables, "ALL FUNCTIONS IN SCHEMA", @@ -3947,7 +3977,8 @@ psql_completion(const char *text, int start, int end) "SEQUENCE", "TABLE", "TABLESPACE", - "TYPE"); + "TYPE", + "VARIABLE"); } else if (TailMatches("GRANT|REVOKE", MatchAny, "ON", "ALL") || TailMatches("REVOKE", "GRANT", "OPTION", "FOR", MatchAny, "ON", "ALL")) @@ -3955,7 +3986,8 @@ psql_completion(const char *text, int start, int end) "PROCEDURES IN SCHEMA", "ROUTINES IN SCHEMA", "SEQUENCES IN SCHEMA", - "TABLES IN SCHEMA"); + "TABLES IN SCHEMA", + "VARIABLES IN SCHEMA"); else if (TailMatches("GRANT|REVOKE", MatchAny, "ON", "FOREIGN") || TailMatches("REVOKE", "GRANT", "OPTION", "FOR", MatchAny, "ON", "FOREIGN")) COMPLETE_WITH("DATA WRAPPER", "SERVER"); @@ -3991,6 +4023,8 @@ psql_completion(const char *text, int start, int end) COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces); else if (TailMatches("TYPE")) COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_datatypes); + else if (TailMatches("VARIABLE")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_variables); else if (TailMatches("GRANT", MatchAny, MatchAny, MatchAny)) COMPLETE_WITH("TO"); else @@ -4275,7 +4309,7 @@ psql_completion(const char *text, int start, int end) /* PREPARE xx AS */ else if (Matches("PREPARE", MatchAny, "AS")) - COMPLETE_WITH("SELECT", "UPDATE", "INSERT INTO", "DELETE FROM"); + COMPLETE_WITH("SELECT", "UPDATE", "INSERT INTO", "DELETE FROM", "LET"); /* * PREPARE TRANSACTION is missing on purpose. It's intended for transaction @@ -4568,6 +4602,14 @@ psql_completion(const char *text, int start, int end) else if (TailMatches("UPDATE", MatchAny, "SET", MatchAnyExcept("*="))) COMPLETE_WITH("="); +/* LET --- can be inside EXPLAIN, PREPARE etc */ + /* If prev. word is LET suggest a list of variables */ + else if (TailMatches("LET")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_variables); + /* Complete LET <variable> with "=" */ + else if (TailMatches("LET", MatchAny)) + COMPLETE_WITH("="); + /* USER MAPPING */ else if (Matches("ALTER|CREATE|DROP", "USER", "MAPPING")) COMPLETE_WITH("FOR"); @@ -4739,6 +4781,8 @@ psql_completion(const char *text, int start, int end) COMPLETE_WITH_QUERY(Query_for_list_of_roles); else if (TailMatchesCS("\\dv*")) COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views); + else if (TailMatchesCS("\\dV*")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_variables); else if (TailMatchesCS("\\dx*")) COMPLETE_WITH_QUERY(Query_for_list_of_extensions); else if (TailMatchesCS("\\dX*")) -- 2.40.1 [text/x-patch] v20230516-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch (19.6K, ../../CAFj8pRDF_eygHWcG8Y9d73zmus=ohVup-ngP7w-m4tbtWg14KQ@mail.gmail.com/5-v20230516-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch) download | inline diff: From 20ea8325c7845c119e80a299a7eb43215f42aa37 Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Mon, 4 Apr 2022 20:49:11 +0200 Subject: [PATCH 07/10] possibility to dump session variables by pg_dump Enhancing pg_dump about session variables support --- src/bin/pg_dump/common.c | 3 +- src/bin/pg_dump/dumputils.c | 6 + src/bin/pg_dump/pg_backup.h | 2 + src/bin/pg_dump/pg_backup_archiver.c | 9 + src/bin/pg_dump/pg_dump.c | 237 ++++++++++++++++++++++++++- src/bin/pg_dump/pg_dump.h | 25 ++- src/bin/pg_dump/pg_dump_sort.c | 6 + src/bin/pg_dump/pg_restore.c | 9 +- src/bin/pg_dump/t/002_pg_dump.pl | 82 +++++++++ src/tools/pgindent/typedefs.list | 1 + 10 files changed, 376 insertions(+), 4 deletions(-) diff --git a/src/bin/pg_dump/common.c b/src/bin/pg_dump/common.c index 5d988986ed..4defc8c216 100644 --- a/src/bin/pg_dump/common.c +++ b/src/bin/pg_dump/common.c @@ -264,7 +264,8 @@ getSchemaData(Archive *fout, int *numTablesPtr) pg_log_info("reading subscriptions"); getSubscriptions(fout); - free(inhinfo); /* not needed any longer */ + pg_log_info("reading variables"); + getVariables(fout); *numTablesPtr = numTables; return tblinfo; diff --git a/src/bin/pg_dump/dumputils.c b/src/bin/pg_dump/dumputils.c index 079693585c..bf628834e3 100644 --- a/src/bin/pg_dump/dumputils.c +++ b/src/bin/pg_dump/dumputils.c @@ -511,6 +511,12 @@ do { \ CONVERT_PRIV('r', "SELECT"); CONVERT_PRIV('w', "UPDATE"); } + else if (strcmp(type, "VARIABLE") == 0 || + strcmp(type, "VARIABLES") == 0) + { + CONVERT_PRIV('r', "SELECT"); + CONVERT_PRIV('w', "UPDATE"); + } else abort(); diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index aba780ef4b..a0274bffcb 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -132,12 +132,14 @@ typedef struct _restoreOptions int selFunction; int selTrigger; int selTable; + int selVariable; SimpleStringList indexNames; SimpleStringList functionNames; SimpleStringList schemaNames; SimpleStringList schemaExcludeNames; SimpleStringList triggerNames; SimpleStringList tableNames; + SimpleStringList variableNames; int useDB; ConnParams cparams; /* parameters to use if useDB */ diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index d518349e10..fbbddb6bd9 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -2971,6 +2971,14 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH) !simple_string_list_member(&ropt->triggerNames, te->tag)) return 0; } + else if (strcmp(te->desc, "VARIABLE") == 0) + { + if (!ropt->selVariable) + return 0; + if (ropt->variableNames.head != NULL && + !simple_string_list_member(&ropt->variableNames, te->tag)) + return 0; + } else return 0; } @@ -3459,6 +3467,7 @@ _getObjectDescription(PQExpBuffer buf, const TocEntry *te) strcmp(type, "TEXT SEARCH DICTIONARY") == 0 || strcmp(type, "TEXT SEARCH CONFIGURATION") == 0 || strcmp(type, "TYPE") == 0 || + strcmp(type, "VARIABLE") == 0 || strcmp(type, "VIEW") == 0 || /* non-schema-specified objects */ strcmp(type, "DATABASE") == 0 || diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 41a51ec5cd..dddf741508 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -296,6 +296,7 @@ static void dumpPolicy(Archive *fout, const PolicyInfo *polinfo); static void dumpPublication(Archive *fout, const PublicationInfo *pubinfo); static void dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo); static void dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo); +static void dumpVariable(Archive *fout, const VariableInfo *varinfo); static void dumpDatabase(Archive *fout); static void dumpDatabaseConfig(Archive *AH, PQExpBuffer outbuf, const char *dbname, Oid dboid); @@ -4923,6 +4924,232 @@ get_next_possible_free_pg_type_oid(Archive *fout, PQExpBuffer upgrade_query) return next_possible_free_oid; } +/* + * getVariables + * get information about variables + */ +void +getVariables(Archive *fout) +{ + PQExpBuffer query; + PGresult *res; + VariableInfo *varinfo; + int i_tableoid; + int i_oid; + int i_varname; + int i_varnamespace; + int i_vartype; + int i_vartypname; + int i_vardefexpr; + int i_vareoxaction; + int i_varisnotnull; + int i_varisimmutable; + int i_varowner; + int i_varcollation; + int i_varacl; + int i_acldefault; + int i, + ntups; + + if (fout->remoteVersion < 160000) + return; + + query = createPQExpBuffer(); + + resetPQExpBuffer(query); + + /* Get the variables in current database. */ + appendPQExpBuffer(query, + "SELECT v.tableoid, v.oid, v.varname,\n" + "v.vareoxaction,\n" + "v.varnamespace,\n" + "v.vartype,\n" + "pg_catalog.format_type(v.vartype, v.vartypmod) as vartypname,\n" + "v.varisnotnull,\n" + "v.varisimmutable,\n" + "CASE WHEN v.varcollation <> t.typcollation " + "THEN v.varcollation ELSE 0 END AS varcollation,\n" + "pg_catalog.pg_get_expr(v.vardefexpr,0) as vardefexpr,\n" + "v.varowner,\n" + "v.varacl,\n" + "acldefault('V', v.varowner) AS acldefault\n" + "FROM pg_catalog.pg_variable v\n" + "JOIN pg_catalog.pg_type t " + "ON (v.vartype = t.oid)"); + + res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK); + + ntups = PQntuples(res); + + i_tableoid = PQfnumber(res, "tableoid"); + i_oid = PQfnumber(res, "oid"); + i_varname = PQfnumber(res, "varname"); + i_varnamespace = PQfnumber(res, "varnamespace"); + i_vartype = PQfnumber(res, "vartype"); + i_vartypname = PQfnumber(res, "vartypname"); + i_vareoxaction = PQfnumber(res, "vareoxaction"); + i_vardefexpr = PQfnumber(res, "vardefexpr"); + i_varisnotnull = PQfnumber(res, "varisnotnull"); + i_varisimmutable = PQfnumber(res, "varisimmutable"); + i_varcollation = PQfnumber(res, "varcollation"); + + i_varowner = PQfnumber(res, "varowner"); + i_varacl = PQfnumber(res, "varacl"); + i_acldefault = PQfnumber(res, "acldefault"); + + varinfo = pg_malloc(ntups * sizeof(VariableInfo)); + + for (i = 0; i < ntups; i++) + { + TypeInfo *vtype; + + varinfo[i].dobj.objType = DO_VARIABLE; + varinfo[i].dobj.catId.tableoid = + atooid(PQgetvalue(res, i, i_tableoid)); + varinfo[i].dobj.catId.oid = atooid(PQgetvalue(res, i, i_oid)); + AssignDumpId(&varinfo[i].dobj); + varinfo[i].dobj.name = pg_strdup(PQgetvalue(res, i, i_varname)); + varinfo[i].dobj.namespace = + findNamespace(atooid(PQgetvalue(res, i, i_varnamespace))); + + varinfo[i].vartype = atooid(PQgetvalue(res, i, i_vartype)); + varinfo[i].vartypname = pg_strdup(PQgetvalue(res, i, i_vartypname)); + varinfo[i].vareoxaction = pg_strdup(PQgetvalue(res, i, i_vareoxaction)); + varinfo[i].varisnotnull = *(PQgetvalue(res, i, i_varisnotnull)) == 't'; + varinfo[i].varisimmutable = *(PQgetvalue(res, i, i_varisimmutable)) == 't'; + varinfo[i].varcollation = atooid(PQgetvalue(res, i, i_varcollation)); + + varinfo[i].dacl.acl = pg_strdup(PQgetvalue(res, i, i_varacl)); + varinfo[i].dacl.acldefault = pg_strdup(PQgetvalue(res, i, i_acldefault)); + varinfo[i].dacl.privtype = 0; + varinfo[i].dacl.initprivs = NULL; + varinfo[i].rolname = getRoleName(PQgetvalue(res, i, i_varowner)); + + /* Decide whether we want to dump it */ + selectDumpableObject(&(varinfo[i].dobj), fout); + + /* Do not try to dump ACL if no ACL exists. */ + if (!PQgetisnull(res, i, i_varacl)) + varinfo[i].dobj.components |= DUMP_COMPONENT_ACL; + + if (PQgetisnull(res, i, i_vardefexpr)) + varinfo[i].vardefexpr = NULL; + else + varinfo[i].vardefexpr = pg_strdup(PQgetvalue(res, i, i_vardefexpr)); + + if (strlen(varinfo[i].rolname) == 0) + pg_log_warning("owner of variable \"%s\" appears to be invalid", + varinfo[i].dobj.name); + + /* Decide whether we want to dump it */ + selectDumpableObject(&(varinfo[i].dobj), fout); + + vtype = findTypeByOid(varinfo[i].vartype); + addObjectDependency(&varinfo[i].dobj, vtype->dobj.dumpId); + } + PQclear(res); + + destroyPQExpBuffer(query); +} + +/* + * dumpVariable + * dump the definition of the given session variable + */ +static void +dumpVariable(Archive *fout, const VariableInfo *varinfo) +{ + DumpOptions *dopt = fout->dopt; + + PQExpBuffer delq; + PQExpBuffer query; + char *qualvarname; + const char *vartypname; + const char *vardefexpr; + const char *vareoxaction; + const char *varisimmutable; + Oid varcollation; + bool varisnotnull; + + /* Skip if not to be dumped */ + if (!varinfo->dobj.dump || dopt->dataOnly) + return; + + delq = createPQExpBuffer(); + query = createPQExpBuffer(); + + qualvarname = pg_strdup(fmtQualifiedDumpable(varinfo)); + vartypname = varinfo->vartypname; + vardefexpr = varinfo->vardefexpr; + vareoxaction = varinfo->vareoxaction; + varisnotnull = varinfo->varisnotnull; + varisimmutable = varinfo->varisimmutable ? "IMMUTABLE " : ""; + varcollation = varinfo->varcollation; + + appendPQExpBuffer(delq, "DROP VARIABLE %s;\n", + qualvarname); + + appendPQExpBuffer(query, "CREATE %sVARIABLE %s AS %s", + varisimmutable, qualvarname, vartypname); + + if (OidIsValid(varcollation)) + { + CollInfo *coll; + + coll = findCollationByOid(varcollation); + if (coll) + appendPQExpBuffer(query, " COLLATE %s", + fmtQualifiedDumpable(coll)); + } + + if (varisnotnull) + appendPQExpBuffer(query, " NOT NULL"); + + if (vardefexpr) + appendPQExpBuffer(query, " DEFAULT %s", + vardefexpr); + + if (strcmp(vareoxaction, "d") == 0) + appendPQExpBuffer(query, " ON COMMIT DROP"); + else if (strcmp(vareoxaction, "r") == 0) + appendPQExpBuffer(query, " ON TRANSACTION END RESET"); + + appendPQExpBuffer(query, ";\n"); + + if (varinfo->dobj.dump & DUMP_COMPONENT_DEFINITION) + ArchiveEntry(fout, varinfo->dobj.catId, varinfo->dobj.dumpId, + ARCHIVE_OPTS(.tag = varinfo->dobj.name, + .namespace = varinfo->dobj.namespace->dobj.name, + .owner = varinfo->rolname, + .description = "VARIABLE", + .section = SECTION_PRE_DATA, + .createStmt = query->data, + .dropStmt = delq->data)); + + /* Dump comment if any */ + if (varinfo->dobj.dump & DUMP_COMPONENT_COMMENT) + dumpComment(fout, "VARIABLE", qualvarname, + NULL, varinfo->rolname, + varinfo->dobj.catId, 0, varinfo->dobj.dumpId); + + /* Dump ACL if any */ + if (varinfo->dobj.dump & DUMP_COMPONENT_ACL) + { + char *qvarname = pg_strdup(fmtId(varinfo->dobj.name)); + + dumpACL(fout, varinfo->dobj.dumpId, InvalidDumpId, "VARIABLE", + qvarname, NULL, + varinfo->dobj.namespace->dobj.name, varinfo->rolname, &varinfo->dacl); + + free(qvarname); + } + + destroyPQExpBuffer(delq); + destroyPQExpBuffer(query); + + free(qualvarname); +} + static void binary_upgrade_set_type_oids_by_type_oid(Archive *fout, PQExpBuffer upgrade_buffer, @@ -9662,7 +9889,8 @@ getAdditionalACLs(Archive *fout) dobj->objType == DO_TABLE || dobj->objType == DO_PROCLANG || dobj->objType == DO_FDW || - dobj->objType == DO_FOREIGN_SERVER) + dobj->objType == DO_FOREIGN_SERVER || + dobj->objType == DO_VARIABLE) { DumpableObjectWithAcl *daobj = (DumpableObjectWithAcl *) dobj; @@ -10252,6 +10480,9 @@ dumpDumpableObject(Archive *fout, DumpableObject *dobj) case DO_SUBSCRIPTION: dumpSubscription(fout, (const SubscriptionInfo *) dobj); break; + case DO_VARIABLE: + dumpVariable(fout, (VariableInfo *) dobj); + break; case DO_PRE_DATA_BOUNDARY: case DO_POST_DATA_BOUNDARY: /* never dumped, nothing to do */ @@ -14649,6 +14880,9 @@ dumpDefaultACL(Archive *fout, const DefaultACLInfo *daclinfo) case DEFACLOBJ_NAMESPACE: type = "SCHEMAS"; break; + case DEFACLOBJ_VARIABLE: + type = "VARIABLES"; + break; default: /* shouldn't get here */ pg_fatal("unrecognized object type in default privileges: %d", @@ -18193,6 +18427,7 @@ addBoundaryDependencies(DumpableObject **dobjs, int numObjs, case DO_CONVERSION: case DO_TABLE: case DO_TABLE_ATTACH: + case DO_VARIABLE: case DO_ATTRDEF: case DO_PROCLANG: case DO_CAST: diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index ed6ce41ad7..ab9d2215fb 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -52,6 +52,7 @@ typedef enum DO_TABLE, DO_TABLE_ATTACH, DO_ATTRDEF, + DO_VARIABLE, DO_INDEX, DO_INDEX_ATTACH, DO_STATSEXT, @@ -82,7 +83,7 @@ typedef enum DO_PUBLICATION, DO_PUBLICATION_REL, DO_PUBLICATION_TABLE_IN_SCHEMA, - DO_SUBSCRIPTION + DO_SUBSCRIPTION, } DumpableObjectType; /* @@ -666,6 +667,27 @@ typedef struct _SubscriptionInfo char *subpasswordrequired; } SubscriptionInfo; +/* + * The VariableInfo struct is used to represent session variables + */ +typedef struct _VariableInfo +{ + DumpableObject dobj; + DumpableAcl dacl; + Oid vartype; + char *vartypname; + char *vareoxaction; + char *vardefexpr; + char *varacl; + char *rvaracl; + char *initvaracl; + char *initrvaracl; + bool varisnotnull; + bool varisimmutable; + Oid varcollation; + const char *rolname; /* name of owner, or empty string */ +} VariableInfo; + /* * common utility functions */ @@ -749,5 +771,6 @@ extern void getPublicationNamespaces(Archive *fout); extern void getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables); extern void getSubscriptions(Archive *fout); +extern void getVariables(Archive *fout); #endif /* PG_DUMP_H */ diff --git a/src/bin/pg_dump/pg_dump_sort.c b/src/bin/pg_dump/pg_dump_sort.c index 745578d855..b1dcf98e64 100644 --- a/src/bin/pg_dump/pg_dump_sort.c +++ b/src/bin/pg_dump/pg_dump_sort.c @@ -76,6 +76,7 @@ enum dbObjectTypePriorities PRIO_DUMMY_TYPE, PRIO_ATTRDEF, PRIO_LARGE_OBJECT, + PRIO_VARIABLE, PRIO_PRE_DATA_BOUNDARY, /* boundary! */ PRIO_TABLE_DATA, PRIO_SEQUENCE_SET, @@ -116,6 +117,7 @@ static const int dbObjectTypePriority[] = PRIO_TABLE, /* DO_TABLE */ PRIO_TABLE_ATTACH, /* DO_TABLE_ATTACH */ PRIO_ATTRDEF, /* DO_ATTRDEF */ + PRIO_VARIABLE, /* DO_VARIABLE */ PRIO_INDEX, /* DO_INDEX */ PRIO_INDEX_ATTACH, /* DO_INDEX_ATTACH */ PRIO_STATSEXT, /* DO_STATSEXT */ @@ -1508,6 +1510,10 @@ describeDumpableObject(DumpableObject *obj, char *buf, int bufsize) "POST-DATA BOUNDARY (ID %d)", obj->dumpId); return; + case DO_VARIABLE: + snprintf(buf, bufsize, + "VARIABLE %s (ID %d OID %u)", + obj->name, obj->dumpId, obj->catId.oid); } /* shouldn't get here */ snprintf(buf, bufsize, diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index 049a100634..830cde5421 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -103,6 +103,7 @@ main(int argc, char **argv) {"trigger", 1, NULL, 'T'}, {"use-list", 1, NULL, 'L'}, {"username", 1, NULL, 'U'}, + {"variable", 1, NULL, 'A'}, {"verbose", 0, NULL, 'v'}, {"single-transaction", 0, NULL, '1'}, @@ -151,7 +152,7 @@ main(int argc, char **argv) } } - while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1", + while ((c = getopt_long(argc, argv, "A:acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1", cmdopts, NULL)) != -1) { switch (c) @@ -159,6 +160,11 @@ main(int argc, char **argv) case 'a': /* Dump data only */ opts->dataOnly = 1; break; + case 'A': /* vAriable */ + opts->selTypes = 1; + opts->selVariable = 1; + simple_string_list_append(&opts->variableNames, optarg); + break; case 'c': /* clean (i.e., drop) schema prior to create */ opts->dropSchema = 1; break; @@ -444,6 +450,7 @@ usage(const char *progname) printf(_("\nOptions controlling the restore:\n")); printf(_(" -a, --data-only restore only the data, no schema\n")); + printf(_(" -A, --variable=NAME restore named session variable\n")); printf(_(" -c, --clean clean (drop) database objects before recreating\n")); printf(_(" -C, --create create the target database\n")); printf(_(" -e, --exit-on-error exit on error, default is to continue\n")); diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 93e24d5145..f60a117995 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -805,6 +805,16 @@ my %tests = ( unlike => { no_privs => 1, }, }, + 'ALTER DEFAULT PRIVILEGES FOR ROLE regress_dump_test_role GRANT SELECT ON VARIABLES TO PUBLIC' + => { + create_order => 56, + create_sql => 'ALTER DEFAULT PRIVILEGES FOR ROLE regress_dump_test_role GRANT SELECT ON VARIABLES TO PUBLIC;', + regexp => qr/^ + \QALTER DEFAULT PRIVILEGES FOR ROLE regress_dump_test_role GRANT SELECT ON VARIABLES TO PUBLIC;\E/xm, + like => { %full_runs, section_post_data => 1, }, + unlike => { no_privs => 1, }, + }, + 'ALTER ROLE regress_dump_test_role' => { regexp => qr/^ \QALTER ROLE regress_dump_test_role WITH \E @@ -1658,6 +1668,23 @@ my %tests = ( }, }, + 'COMMENT ON VARIABLE dump_test.variable1' => { + create_order => 71, + create_sql => 'COMMENT ON VARIABLE dump_test.variable1 + IS \'comment on variable\';', + regexp => + qr/^\QCOMMENT ON VARIABLE dump_test.variable1 IS 'comment on variable';\E/m, + like => { + %full_runs, + %dump_test_schema_runs, + section_pre_data => 1, + }, + unlike => { + exclude_dump_test_schema => 1, + only_dump_measurement => 1, + }, + }, + 'COPY test_table' => { create_order => 4, create_sql => 'INSERT INTO dump_test.test_table (col1) ' @@ -3852,6 +3879,42 @@ my %tests = ( }, }, + 'CREATE VARIABLE test_variable' => { + all_runs => 1, + catch_all => 'CREATE ... commands', + create_order => 61, + create_sql => 'CREATE VARIABLE dump_test.variable1 AS integer DEFAULT 0;', + regexp => qr/^ + \QCREATE VARIABLE dump_test.variable1 AS integer DEFAULT 0;\E/xm, + like => { + %full_runs, + %dump_test_schema_runs, + section_pre_data => 1, + }, + unlike => { + exclude_dump_test_schema => 1, + only_dump_measurement => 1, + }, + }, + + 'CREATE IMMUTABLE VARIABLE test_variable' => { + all_runs => 1, + catch_all => 'CREATE ... commands', + create_order => 61, + create_sql => 'CREATE IMMUTABLE VARIABLE dump_test.variable2 AS integer DEFAULT 0;', + regexp => qr/^ + \QCREATE IMMUTABLE VARIABLE dump_test.variable2 AS integer DEFAULT 0;\E/xm, + like => { + %full_runs, + %dump_test_schema_runs, + section_pre_data => 1, + }, + unlike => { + exclude_dump_test_schema => 1, + only_dump_measurement => 1, + }, + }, + 'CREATE VIEW test_view' => { create_order => 61, create_sql => 'CREATE VIEW dump_test.test_view @@ -4312,6 +4375,25 @@ my %tests = ( like => {}, }, + 'GRANT SELECT ON VARIABLE dump_test.variable1' => { + create_order => 73, + create_sql => + 'GRANT SELECT ON VARIABLE dump_test.variable1 TO regress_dump_test_role;', + regexp => qr/^ + \QGRANT SELECT ON VARIABLE dump_test.variable1 TO regress_dump_test_role;\E + /xm, + like => { + %full_runs, + %dump_test_schema_runs, + section_pre_data => 1, + }, + unlike => { + exclude_dump_test_schema => 1, + no_privs => 1, + only_dump_measurement => 1, + }, + }, + 'REFRESH MATERIALIZED VIEW matview' => { regexp => qr/^\QREFRESH MATERIALIZED VIEW dump_test.matview;\E/m, like => diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 7e2fe9f36c..b1f1ad3b24 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2942,6 +2942,7 @@ Variable VariableAssignHook VariableCache VariableCacheData +VariableInfo VariableSetKind VariableSetStmt VariableShowStmt -- 2.40.1 [text/x-patch] v20230516-1-0008-regress-tests-for-session-variables.patch (64.7K, ../../CAFj8pRDF_eygHWcG8Y9d73zmus=ohVup-ngP7w-m4tbtWg14KQ@mail.gmail.com/6-v20230516-1-0008-regress-tests-for-session-variables.patch) download | inline diff: From 0660b89d37a782a2cbd540ebda6712910efe2cd0 Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Mon, 4 Apr 2022 20:54:43 +0200 Subject: [PATCH 08/10] regress tests for session variables Checks of basic functionality, check of usage session variables in subtransactions and from PLpgSQL --- .../isolation/expected/session-variable.out | 112 ++ src/test/isolation/isolation_schedule | 1 + .../isolation/specs/session-variable.spec | 53 + .../regress/expected/session_variables.out | 1564 +++++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/sql/session_variables.sql | 1203 +++++++++++++ 6 files changed, 2934 insertions(+), 1 deletion(-) create mode 100644 src/test/isolation/expected/session-variable.out create mode 100644 src/test/isolation/specs/session-variable.spec create mode 100644 src/test/regress/expected/session_variables.out create mode 100644 src/test/regress/sql/session_variables.sql diff --git a/src/test/isolation/expected/session-variable.out b/src/test/isolation/expected/session-variable.out new file mode 100644 index 0000000000..1c4149f6ef --- /dev/null +++ b/src/test/isolation/expected/session-variable.out @@ -0,0 +1,112 @@ +Parsed test spec with 4 sessions + +starting permutation: let val drop val +step let: LET myvar = 'test'; +step val: SELECT myvar; +myvar +----- +test +(1 row) + +step drop: DROP VARIABLE myvar; +step val: SELECT myvar; +ERROR: column or variable "myvar" does not exist + +starting permutation: let val s1 drop val sr1 +step let: LET myvar = 'test'; +step val: SELECT myvar; +myvar +----- +test +(1 row) + +step s1: BEGIN; +step drop: DROP VARIABLE myvar; +step val: SELECT myvar; +ERROR: column or variable "myvar" does not exist +step sr1: ROLLBACK; + +starting permutation: let val dbg drop create dbg val +step let: LET myvar = 'test'; +step val: SELECT myvar; +myvar +----- +test +(1 row) + +step dbg: SELECT schema, name, removed FROM pg_session_variables(); +schema|name |removed +------+-----+------- +public|myvar|f +(1 row) + +step drop: DROP VARIABLE myvar; +step create: CREATE VARIABLE myvar AS text; +step dbg: SELECT schema, name, removed FROM pg_session_variables(); +schema|name|removed +------+----+------- +(0 rows) + +step val: SELECT myvar; +myvar +----- + +(1 row) + + +starting permutation: let val s1 dbg drop create dbg val sr1 +step let: LET myvar = 'test'; +step val: SELECT myvar; +myvar +----- +test +(1 row) + +step s1: BEGIN; +step dbg: SELECT schema, name, removed FROM pg_session_variables(); +schema|name |removed +------+-----+------- +public|myvar|f +(1 row) + +step drop: DROP VARIABLE myvar; +step create: CREATE VARIABLE myvar AS text; +step dbg: SELECT schema, name, removed FROM pg_session_variables(); +schema|name |removed +------+-----+------- +public|myvar|f +(1 row) + +step val: SELECT myvar; +myvar +----- + +(1 row) + +step sr1: ROLLBACK; + +starting permutation: create3 let3 s3 o_c_d o_eox_r create4 let4 drop4 drop3 inval3 discard sc3 clean state +step create3: CREATE VARIABLE myvar3 AS text; +step let3: LET myvar3 = 'test'; +step s3: BEGIN; +step o_c_d: CREATE TEMP VARIABLE myvar_o_c_d AS text ON COMMIT DROP; +step o_eox_r: CREATE VARIABLE myvar_o_eox_r AS text ON TRANSACTION END RESET; LET myvar_o_eox_r = 'test'; +step create4: CREATE VARIABLE myvar4 AS text; +step let4: LET myvar4 = 'test'; +step drop4: DROP VARIABLE myvar4; +step drop3: DROP VARIABLE myvar3; +step inval3: SELECT COUNT(*) >= 0 FROM pg_foreign_table; +?column? +-------- +t +(1 row) + +step discard: DISCARD VARIABLES; +step sc3: COMMIT; +step clean: DROP VARIABLE myvar_o_eox_r; +step state: SELECT varname FROM pg_variable; +varname +------- +myvar +(1 row) + diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index 4fc56ae99c..809f47b941 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -110,3 +110,4 @@ test: serializable-parallel test: serializable-parallel-2 test: serializable-parallel-3 test: matview-write-skew +test: session-variable diff --git a/src/test/isolation/specs/session-variable.spec b/src/test/isolation/specs/session-variable.spec new file mode 100644 index 0000000000..5d089c8a4e --- /dev/null +++ b/src/test/isolation/specs/session-variable.spec @@ -0,0 +1,53 @@ +# Test session variables memory cleanup for sinval + +setup +{ + CREATE VARIABLE myvar AS text; +} + +teardown +{ + DROP VARIABLE IF EXISTS myvar; +} + +session s1 +step s1 { BEGIN; } +step let { LET myvar = 'test'; } +step val { SELECT myvar; } +step dbg { SELECT schema, name, removed FROM pg_session_variables(); } +step sr1 { ROLLBACK; } + +session s2 +step drop { DROP VARIABLE myvar; } +step create { CREATE VARIABLE myvar AS text; } + +session s3 +step s3 { BEGIN; } +step let3 { LET myvar3 = 'test'; } +step o_c_d { CREATE TEMP VARIABLE myvar_o_c_d AS text ON COMMIT DROP; } +step o_eox_r { CREATE VARIABLE myvar_o_eox_r AS text ON TRANSACTION END RESET; LET myvar_o_eox_r = 'test'; } +step create4 { CREATE VARIABLE myvar4 AS text; } +step let4 { LET myvar4 = 'test'; } +step drop4 { DROP VARIABLE myvar4; } +step inval3 { SELECT COUNT(*) >= 0 FROM pg_foreign_table; } +step discard { DISCARD VARIABLES; } +step sc3 { COMMIT; } +step clean { DROP VARIABLE myvar_o_eox_r; } +step state { SELECT varname FROM pg_variable; } + +session s4 +step create3 { CREATE VARIABLE myvar3 AS text; } +step drop3 { DROP VARIABLE myvar3; } + +# Concurrent drop of a known variable should lead to an error +permutation let val drop val +# Same, but with an explicit transaction +permutation let val s1 drop val sr1 +# Concurrent drop/create of a known variable should lead to empty variable +permutation let val dbg drop create dbg val +# Concurrent drop/create of a known variable should lead to empty variable +# We need a transaction to make sure that we won't accept invalidation when +# calling the dbg step after the concurrent drop +permutation let val s1 dbg drop create dbg val sr1 +# test for DISCARD ALL when all internal queues have actions registered +permutation create3 let3 s3 o_c_d o_eox_r create4 let4 drop4 drop3 inval3 discard sc3 clean state diff --git a/src/test/regress/expected/session_variables.out b/src/test/regress/expected/session_variables.out new file mode 100644 index 0000000000..463b2c96f6 --- /dev/null +++ b/src/test/regress/expected/session_variables.out @@ -0,0 +1,1564 @@ +-- should fail, pseudotypes are not allowed +CREATE VARIABLE xx AS anyelement; +ERROR: session variable cannot be pseudo-type anyelement +-- should be ok +CREATE SCHEMA svartest CREATE VARIABLE xx AS int CREATE TABLE foo(a int); +LET svartest.xx = 100; +DROP SCHEMA svartest CASCADE; +NOTICE: drop cascades to 2 other objects +DETAIL: drop cascades to table svartest.foo +drop cascades to session variable svartest.xx +-- should fail +LET svartest.xx = 100; +ERROR: session variable "svartest.xx" doesn't exist +LINE 1: LET svartest.xx = 100; + ^ +CREATE SCHEMA svartest; +SET search_path = svartest; +CREATE VARIABLE var1 AS integer; +CREATE TEMP VARIABLE var2 AS text; +DROP VARIABLE var1, var2; +-- functional interface +CREATE VARIABLE var1 AS numeric; +CREATE ROLE var_test_role; +GRANT USAGE ON SCHEMA svartest TO var_test_role; +SET ROLE TO var_test_role; +-- should fail +SELECT var1; +ERROR: permission denied for session variable var1 +SET ROLE TO DEFAULT; +GRANT SELECT ON VARIABLE var1 TO var_test_role; +SET ROLE TO var_test_role; +-- should fail +LET var1 = 10; +ERROR: permission denied for session variable var1 +-- should work +SELECT var1; + var1 +------ + +(1 row) + +SET ROLE TO DEFAULT; +GRANT UPDATE ON VARIABLE var1 TO var_test_role; +SET ROLE TO var_test_role; +-- should work +LET var1 = 333; +SET ROLE TO DEFAULT; +REVOKE ALL ON VARIABLE var1 FROM var_test_role; +CREATE OR REPLACE FUNCTION secure_var() +RETURNS int AS $$ + SELECT svartest.var1::int; +$$ LANGUAGE sql SECURITY DEFINER; +SELECT secure_var(); + secure_var +------------ + 333 +(1 row) + +SET ROLE TO var_test_role; +-- should fail +SELECT svartest.var1; +ERROR: permission denied for session variable var1 +-- should work; +SELECT secure_var(); + secure_var +------------ + 333 +(1 row) + +SET ROLE TO DEFAULT; +EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM generate_series(1,100) g(v) WHERE v = var1; + QUERY PLAN +----------------------------------------------- + Function Scan on pg_catalog.generate_series g + Output: v + Function Call: generate_series(1, 100) + Filter: ((g.v)::numeric = var1) +(4 rows) + +CREATE VIEW schema_var_view AS SELECT var1; +SELECT * FROM schema_var_view; + var1 +------ + 333 +(1 row) + +\c - +SET search_path = svartest; +-- should work still, but var will be empty +SELECT * FROM schema_var_view; + var1 +------ + +(1 row) + +LET var1 = pi(); +SELECT var1; + var1 +------------------ + 3.14159265358979 +(1 row) + +-- we can see execution plan of LET statement +EXPLAIN (VERBOSE, COSTS OFF) LET var1 = pi(); + QUERY PLAN +---------------------------- + SET SESSION VARIABLE + Result + Output: 3.14159265358979 +(3 rows) + +SELECT var1; + var1 +------------------ + 3.14159265358979 +(1 row) + +CREATE VARIABLE var3 AS int; +CREATE OR REPLACE FUNCTION inc(int) +RETURNS int AS $$ +BEGIN + LET svartest.var3 = COALESCE(svartest.var3 + $1, $1); + RETURN var3; +END; +$$ LANGUAGE plpgsql; +SELECT inc(1); + inc +----- + 1 +(1 row) + +SELECT inc(1); + inc +----- + 2 +(1 row) + +SELECT inc(1); + inc +----- + 3 +(1 row) + +SELECT inc(1) FROM generate_series(1,10); + inc +----- + 4 + 5 + 6 + 7 + 8 + 9 + 10 + 11 + 12 + 13 +(10 rows) + +SET ROLE TO var_test_role; +-- should fail +LET var3 = 0; +ERROR: permission denied for session variable var3 +SET ROLE TO DEFAULT; +DROP VIEW schema_var_view; +DROP VARIABLE var1 CASCADE; +DROP VARIABLE var3 CASCADE; +-- composite variables +CREATE TYPE sv_xyz AS (x int, y int, z numeric(10,2)); +CREATE VARIABLE v1 AS sv_xyz; +CREATE VARIABLE v2 AS sv_xyz; +\d v1 +\d v2 +LET v1 = (1,2,3.14); +LET v2 = (10,20,3.14*10); +-- should work too - there are prepared casts +LET v1 = (1,2,3.14); +SELECT v1; + v1 +------------ + (1,2,3.14) +(1 row) + +SELECT v2; + v2 +--------------- + (10,20,31.40) +(1 row) + +SELECT (v1).*; + x | y | z +---+---+------ + 1 | 2 | 3.14 +(1 row) + +SELECT (v2).*; + x | y | z +----+----+------- + 10 | 20 | 31.40 +(1 row) + +SELECT v1.x + v1.z; + ?column? +---------- + 4.14 +(1 row) + +SELECT v2.x + v2.z; + ?column? +---------- + 41.40 +(1 row) + +-- access to composite fields should be safe too +-- should fail +SET ROLE TO var_test_role; +SELECT v2.x; +ERROR: permission denied for session variable v2 +SET ROLE TO DEFAULT; +DROP VARIABLE v1; +DROP VARIABLE v2; +REVOKE USAGE ON SCHEMA svartest FROM var_test_role; +DROP ROLE var_test_role; +-- scalar variables should not be in conflict with qualified column +CREATE VARIABLE varx AS text; +SELECT varx.relname FROM pg_class varx WHERE varx.relname = 'pg_class'; + relname +---------- + pg_class +(1 row) + +-- should fail +SELECT varx.xxx; +ERROR: variable "svartest.varx" is of type "text", which is not a composite type +LINE 1: SELECT varx.xxx; + ^ +-- don't allow multi column query +CREATE TYPE vartesttp AS (a1 int, b1 int, c1 int); +CREATE VARIABLE v1 AS vartesttp; +-- should fail +LET v1 = (NULL::vartesttp).*; +ERROR: assignment expression returned 3 columns +LINE 1: LET v1 = (NULL::vartesttp).*; + ^ +DROP VARIABLE v1; +DROP TYPE vartesttp; +-- variables can be updated under RO transaction +BEGIN; +SET TRANSACTION READ ONLY; +LET varx = 'hello'; +COMMIT; +SELECT varx; + varx +------- + hello +(1 row) + +DROP VARIABLE varx; +CREATE TYPE t1 AS (a int, b numeric, c text); +CREATE VARIABLE v1 AS t1; +LET v1 = (1, pi(), 'hello'); +SELECT v1; + v1 +---------------------------- + (1,3.14159265358979,hello) +(1 row) + +LET v1.b = 10.2222; +SELECT v1; + v1 +------------------- + (1,10.2222,hello) +(1 row) + +-- should fail +LET v1.x = 10; +ERROR: cannot assign to field "x" of column "v1" because there is no such column in data type t1 +LINE 1: LET v1.x = 10; + ^ +DROP VARIABLE v1; +DROP TYPE t1; +-- arrays are supported +CREATE VARIABLE va1 AS numeric[]; +LET va1 = ARRAY[1.1,2.1]; +LET va1[1] = 10.1; +SELECT va1; + va1 +------------ + {10.1,2.1} +(1 row) + +-- LET target doesn't allow srf, should fail +LET va1[generate_series(1,3)] = 100; +ERROR: set-returning functions are not allowed in LET target expression +LINE 1: LET va1[generate_series(1,3)] = 100; + ^ +CREATE TYPE ta2 AS (a numeric, b numeric[]); +CREATE VARIABLE va2 AS ta2; +LET va2 = (10.1, ARRAY[0.0, 0.0]); +LET va2.a = 10.2; +SELECT va2; + va2 +-------------------- + (10.2,"{0.0,0.0}") +(1 row) + +LET va2.b[1] = 10.3; +SELECT va2; + va2 +--------------------- + (10.2,"{10.3,0.0}") +(1 row) + +DROP VARIABLE va1; +DROP VARIABLE va2; +DROP TYPE ta2; +-- default values +CREATE VARIABLE v1 AS numeric DEFAULT pi(); +LET v1 = v1 * 2; +SELECT v1; + v1 +------------------ + 6.28318530717958 +(1 row) + +CREATE TYPE t2 AS (a numeric, b text); +CREATE VARIABLE v2 AS t2 DEFAULT (NULL, 'Hello'); +LET svartest.v2.a = pi(); +SELECT v2; + v2 +-------------------------- + (3.14159265358979,Hello) +(1 row) + +-- should fail due dependency +DROP TYPE t2; +ERROR: cannot drop type t2 because other objects depend on it +DETAIL: session variable v2 depends on type t2 +HINT: Use DROP ... CASCADE to drop the dependent objects too. +-- should be ok +DROP VARIABLE v1; +DROP VARIABLE v2; +-- tests of alters +CREATE SCHEMA var_schema1; +CREATE SCHEMA var_schema2; +CREATE VARIABLE var_schema1.var1 AS integer; +LET var_schema1.var1 = 1000; +SELECT var_schema1.var1; + var1 +------ + 1000 +(1 row) + +ALTER VARIABLE var_schema1.var1 SET SCHEMA var_schema2; +SELECT var_schema2.var1; + var1 +------ + 1000 +(1 row) + +CREATE ROLE var_test_role; +ALTER VARIABLE var_schema2.var1 OWNER TO var_test_role; +SET ROLE TO var_test_role; +-- should fail, no access to schema var_schema2.var +SELECT var_schema2.var1; +ERROR: permission denied for schema var_schema2 +DROP VARIABLE var_schema2.var1; +ERROR: permission denied for schema var_schema2 +SET ROLE TO DEFAULT; +ALTER VARIABLE var_schema2.var1 SET SCHEMA public; +SET ROLE TO var_test_role; +SELECT public.var1; + var1 +------ + 1000 +(1 row) + +ALTER VARIABLE public.var1 RENAME TO var1_renamed; +SELECT public.var1_renamed; + var1_renamed +-------------- + 1000 +(1 row) + +DROP VARIABLE public.var1_renamed; +SET ROLE TO DEFAULt; +-- default rights test +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON VARIABLES TO var_test_role; +CREATE VARIABLE public.var2 AS int; +SET ROLE TO var_test_role; +-- should be ok +LET public.var2 = 100; +SELECT public.var2; + var2 +------ + 100 +(1 row) + +SET ROLE TO DEFAULt; +DROP VARIABLE public.var2; +DROP OWNED BY var_test_role; +DROP ROLE var_test_role; +CREATE VARIABLE xx AS text DEFAULT 'hello'; +SELECT xx, upper(xx); + xx | upper +-------+------- + hello | HELLO +(1 row) + +LET xx = 'Hi'; +SELECT xx; + xx +---- + Hi +(1 row) + +DROP VARIABLE xx; +-- ON TRANSACTION END RESET tests +CREATE VARIABLE t1 AS int DEFAULT -1 ON TRANSACTION END RESET; +BEGIN; + SELECT t1; + t1 +---- + -1 +(1 row) + + LET t1 = 100; + SELECT t1; + t1 +----- + 100 +(1 row) + +COMMIT; +SELECT t1; + t1 +---- + -1 +(1 row) + +BEGIN; + SELECT t1; + t1 +---- + -1 +(1 row) + + LET t1 = 100; + SELECT t1; + t1 +----- + 100 +(1 row) + +ROLLBACK; +SELECT t1; + t1 +---- + -1 +(1 row) + +DROP VARIABLE t1; +CREATE VARIABLE v1 AS int DEFAULT 0; +CREATE VARIABLE v2 AS text DEFAULT 'none'; +LET v1 = 100; +LET v2 = 'Hello'; +SELECT v1, v2; + v1 | v2 +-----+------- + 100 | Hello +(1 row) + +LET v1 = DEFAULT; +LET v2 = DEFAULT; +SELECT v1, v2; + v1 | v2 +----+------ + 0 | none +(1 row) + +DROP VARIABLE v1; +DROP VARIABLE v2; +-- ON COMMIT DROP tests +-- should be 0 always +SELECT count(*) FROM pg_variable; + count +------- + 0 +(1 row) + +CREATE TEMP VARIABLE g AS int ON COMMIT DROP; +SELECT count(*) FROM pg_variable; + count +------- + 0 +(1 row) + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; +COMMIT; +SELECT count(*) FROM pg_variable; + count +------- + 0 +(1 row) + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; +ROLLBACK; +SELECT count(*) FROM pg_variable; + count +------- + 0 +(1 row) + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; + LET g = 1; + DISCARD VARIABLES; +COMMIT; +SELECT count(*) FROM pg_variable; + count +------- + 0 +(1 row) + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; + LET g = 1; + DISCARD VARIABLES; +ROLLBACK; +SELECT count(*) FROM pg_variable; + count +------- + 0 +(1 row) + +-- Encourage use of parallel plans +SET parallel_setup_cost = 0; +SET parallel_tuple_cost = 0; +SET min_parallel_table_scan_size = 0; +SET max_parallel_workers_per_gather = 2; +-- test on query with workers +CREATE TABLE svar_test(a int); +INSERT INTO svar_test SELECT * FROM generate_series(1,1000); +ANALYZE svar_test; +CREATE VARIABLE zero int; +LET zero = 0; +-- parallel workers should be used +EXPLAIN (costs off) SELECT count(*) FROM svar_test WHERE a%10 = zero; + QUERY PLAN +-------------------------------------------- + Aggregate + -> Gather + Workers Planned: 2 + -> Parallel Seq Scan on svar_test + Filter: ((a % 10) = zero) +(5 rows) + +-- result should be 100 +SELECT count(*) FROM svar_test WHERE a%10 = zero; + count +------- + 100 +(1 row) + +LET zero = (SELECT count(*) FROM svar_test); +-- result should be 1000 +SELECT zero; + zero +------ + 1000 +(1 row) + +-- parallel workers should be used +EXPLAIN (costs off) LET zero = (SELECT count(*) FROM svar_test); + QUERY PLAN +---------------------------------------------------------- + SET SESSION VARIABLE + Result + InitPlan 1 (returns $1) + -> Finalize Aggregate + -> Gather + Workers Planned: 2 + -> Partial Aggregate + -> Parallel Seq Scan on svar_test +(8 rows) + +-- test for dependency on relation +CREATE VARIABLE v_table AS svar_test; +ALTER TABLE svar_test ALTER COLUMN a TYPE text; +ERROR: cannot alter table "svar_test" because session variable "svartest.v_table" uses it +DROP VARIABLE v_table; +DROP TABLE svar_test; +DROP VARIABLE zero; +RESET parallel_setup_cost; +RESET parallel_tuple_cost; +RESET min_parallel_table_scan_size; +RESET max_parallel_workers_per_gather; +-- use variables in prepared statements +CREATE VARIABLE v AS numeric; +LET v = 3.14; +-- use variables in views +CREATE VIEW vv AS SELECT COALESCE(v, 0) + 1000 AS result; +SELECT * FROM vv; + result +--------- + 1003.14 +(1 row) + +-- start a new session +\c +SET search_path to svartest; +SELECT * FROM vv; + result +-------- + 1000 +(1 row) + +LET v = 3.14; +SELECT * FROM vv; + result +--------- + 1003.14 +(1 row) + +-- should fail, dependency +DROP VARIABLE v; +ERROR: cannot drop session variable v because other objects depend on it +DETAIL: view vv depends on session variable v +HINT: Use DROP ... CASCADE to drop the dependent objects too. +-- should be ok +DROP VARIABLE v CASCADE; +NOTICE: drop cascades to view vv +-- other features +CREATE VARIABLE dt AS integer DEFAULT 0; +LET dt = 100; +SELECT dt; + dt +----- + 100 +(1 row) + +DISCARD VARIABLES; +SELECT dt; + dt +---- + 0 +(1 row) + +DROP VARIABLE dt; +-- NOT NULL +CREATE VARIABLE v1 AS int NOT NULL; +CREATE VARIABLE v2 AS int NOT NULL DEFAULT NULL; +-- should fail +SELECT v1; +ERROR: null value is not allowed for NOT NULL session variable "svartest.v1" +DETAIL: The session variable was not initialized yet. +SELECT v2; +ERROR: null value is not allowed for NOT NULL session variable "svartest.v2" +LET v1 = NULL; +ERROR: null value is not allowed for NOT NULL session variable "svartest.v1" +LET v2 = NULL; +ERROR: null value is not allowed for NOT NULL session variable "svartest.v2" +LET v1 = DEFAULT; +ERROR: null value is not allowed for NOT NULL session variable "svartest.v1" +LET v2 = DEFAULT; +ERROR: null value is not allowed for NOT NULL session variable "svartest.v2" +-- should be ok +LET v1 = 100; +LET v2 = 1000; +SELECT v1, v2; + v1 | v2 +-----+------ + 100 | 1000 +(1 row) + +DROP VARIABLE v1; +DROP VARIABLE v2; +CREATE VARIABLE tv AS int; +CREATE VARIABLE IF NOT EXISTS tv AS int; +NOTICE: session variable "tv" already exists, skipping +DROP VARIABLE tv; +CREATE IMMUTABLE VARIABLE iv AS int DEFAULT 100; +SELECT iv; + iv +----- + 100 +(1 row) + +-- should fail; +LET iv = 10000; +ERROR: session variable "svartest.iv" is declared IMMUTABLE +DROP VARIABLE iv; +-- different order +CREATE IMMUTABLE VARIABLE iv AS int DEFAULT 100; +-- should to fail +LET iv = 10000; +ERROR: session variable "svartest.iv" is declared IMMUTABLE +-- should be ok +SELECT iv; + iv +----- + 100 +(1 row) + +DROP VARIABLE iv; +CREATE IMMUTABLE VARIABLE iv AS int; +-- should be ok +LET iv = NULL; +-- should fail +LET iv = NULL; +ERROR: session variable "svartest.iv" is declared IMMUTABLE +DROP VARIABLE iv; +-- create variable inside plpgsql block +DO $$ +BEGIN + CREATE VARIABLE do_test_svar AS date DEFAULT '2000-01-01'; +END; +$$; +SELECT do_test_svar; + do_test_svar +-------------- + 01-01-2000 +(1 row) + +DROP VARIABLE do_test_svar; +-- should fail +CREATE IMMUTABLE VARIABLE xx AS int NOT NULL; +ERROR: IMMUTABLE NOT NULL variable requires default expression +-- REASSIGN OWNED test +CREATE ROLE var_test_role1; +CREATE ROLE var_test_role2; +CREATE VARIABLE xxx_var AS int; +ALTER VARIABLE xxx_var OWNER TO var_test_role1; +REASSIGN OWNED BY var_test_role1 to var_test_role2; +SELECT varowner::regrole FROM pg_variable WHERE varname = 'xxx_var'; + varowner +---------------- + var_test_role2 +(1 row) + +DROP OWNED BY var_test_role1; +DROP ROLE var_test_role1; +SELECT count(*) FROM pg_variable WHERE varname = 'xxx_var'; + count +------- + 1 +(1 row) + +DROP OWNED BY var_test_role2; +DROP ROLE var_test_role2; +SELECT count(*) FROM pg_variable WHERE varname = 'xxx_var'; + count +------- + 0 +(1 row) + +-- creating, dropping temporary variable +BEGIN; +CREATE TEMP VARIABLE tempvar AS INT ON COMMIT DROP; +LET tempvar = 100; +SAVEPOINT s1; +DROP VARIABLE tempvar; +ROLLBACK TO s1; +SELECT tempvar; + tempvar +--------- + 100 +(1 row) + +COMMIT; +-- should to fail +LET tempvar = 100; +ERROR: session variable "tempvar" doesn't exist +LINE 1: LET tempvar = 100; + ^ +BEGIN; +SAVEPOINT s1; +CREATE TEMP VARIABLE tempvar AS INT ON COMMIT DROP; +LET tempvar = 100; +ROLLBACK TO s1; +COMMIT; +-- should to fail +LET tempvar = 100; +ERROR: session variable "tempvar" doesn't exist +LINE 1: LET tempvar = 100; + ^ +CREATE VARIABLE var1 AS int; +LET var1 = 100; +BEGIN; +DROP VARIABLE var1; +ROLLBACK; +SELECT var1; + var1 +------ + 100 +(1 row) + +DROP VARIABLE var1; +CREATE VARIABLE var1 AS int DEFAULT 100; +COMMENT ON VARIABLE var1 IS 'some variable comment'; +SELECT pg_catalog.obj_description(oid, 'pg_variable') FROM pg_variable WHERE varname = 'var1'; + obj_description +----------------------- + some variable comment +(1 row) + +DROP VARIABLE var1; +CREATE TABLE xxtab(avar int); +CREATE TYPE xxtype AS (avar int); +CREATE VARIABLE xxtab AS xxtype; +INSERT INTO xxtab VALUES(10); +-- it is ambiguous, but columns are preferred +-- variable is shadowed +SELECT xxtab.avar FROM xxtab; + avar +------ + 10 +(1 row) + +SET session_variables_ambiguity_warning TO on; +SELECT xxtab.avar FROM xxtab; +WARNING: session variable "xxtab.avar" is shadowed +LINE 1: SELECT xxtab.avar FROM xxtab; + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. + avar +------ + 10 +(1 row) + +SET search_path = svartest; +CREATE VARIABLE testvar as int; +-- plpgsql variables are preferred against session variables +DO $$ +<<myblock>> +DECLARE testvar int; +BEGIN + -- should be ok without warning + LET testvar = 100; + -- should be ok without warning + testvar := 1000; + -- should be ok without warning + RAISE NOTICE 'session variable is %', svartest.testvar; + -- should be ok without warning + RAISE NOTICE 'plpgsql variable is %', myblock.testvar; + -- should to print plpgsql variable with warning + RAISE NOTICE 'variable is %', testvar; +END; +$$; +WARNING: session variable "testvar" is shadowed +LINE 1: testvar := 1000 + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. +QUERY: testvar := 1000 +NOTICE: session variable is 100 +NOTICE: plpgsql variable is 1000 +WARNING: session variable "testvar" is shadowed +LINE 1: testvar + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. +QUERY: testvar +NOTICE: variable is 1000 +DROP VARIABLE testvar; +SET session_variables_ambiguity_warning TO default; +-- should be ok +SELECT avar FROM xxtab; + avar +------ + 10 +(1 row) + +CREATE VARIABLE public.avar AS int; +-- should be ok, see the table +SELECT avar FROM xxtab; + avar +------ + 10 +(1 row) + +-- should be ok +SELECT public.avar FROM xxtab; + avar +------ + +(1 row) + +DROP VARIABLE xxtab; +SELECT xxtab.avar FROM xxtab; + avar +------ + 10 +(1 row) + +DROP VARIABLE public.avar; +DROP TYPE xxtype; +DROP TABLE xxtab; +-- test of plan cache invalidation +CREATE VARIABLE xx AS int; +SET plan_cache_mode = force_generic_plan; +PREPARE pp AS SELECT xx; +EXECUTE pp; + xx +---- + +(1 row) + +DROP VARIABLE xx; +CREATE VARIABLE xx AS int; +-- should to work +EXECUTE pp; + xx +---- + +(1 row) + +DROP VARIABLE xx; +DEALLOCATE pp; +SET plan_cache_mode = DEFAULT; +CREATE ROLE var_test_role; +CREATE SCHEMA vartest; +GRANT USAGE ON SCHEMA vartest TO var_test_role; +CREATE VARIABLE vartest.x AS int; +CREATE VARIABLE vartest.y AS int; +LET vartest.x = 100; +LET vartest.y = 101; +GRANT SELECT ON ALL VARIABLES IN SCHEMA vartest TO var_test_role; +SET ROLE TO var_test_role; +SELECT vartest.x, vartest.y; + x | y +-----+----- + 100 | 101 +(1 row) + +SET ROLE TO DEFAULT; +REVOKE SELECT ON ALL VARIABLES IN SCHEMA vartest FROM var_test_role; +SET ROLE TO var_test_role; +-- should to fail +SELECT vartest.x; +ERROR: permission denied for session variable x +SELECT vartest.y; +ERROR: permission denied for session variable y +SET ROLE TO DEFAULT; +DROP VARIABLE vartest.x, vartest.y; +DROP SCHEMA vartest; +DROP ROLE var_test_role; +-- test cached plan +CREATE VARIABLE v1 AS text; +CREATE VARIABLE v2 AS int; +CREATE VARIABLE v3 AS int; +LET v1 = 'test'; +LET v2 = 10; +LET v3 = 5; +PREPARE q1 AS SELECT v1 || i FROM generate_series(1, v2) g(i) WHERE i IN (v2, v3); +SET plan_cache_mode to force_generic_plan; +EXECUTE q1; + ?column? +---------- + test5 + test10 +(2 rows) + +EXPLAIN EXECUTE q1; + QUERY PLAN +----------------------------------------------------------------------- + Function Scan on generate_series g (cost=0.00..0.14 rows=2 width=32) + Filter: (i = ANY (ARRAY[v2, v3])) +(2 rows) + +-- dependecy check +DROP VARIABLE v3; +-- recreate v3 again +CREATE VARIABLE v3 AS int DEFAULT 6; +-- should to work, the plan should be recreated +EXECUTE q1; + ?column? +---------- + test6 + test10 +(2 rows) + +DEALLOCATE q1; +-- fill v1 by long text +LET v1 = repeat(' ', 10000); +PREPARE q1 AS SELECT length(v1); +EXECUTE q1; + length +-------- + 10000 +(1 row) + +LET v1 = repeat(' ', 5000); +EXECUTE q1; + length +-------- + 5000 +(1 row) + +DEALLOCATE q1; +SET plan_cache_mode to default; +DROP VARIABLE v1, v2, v3; +CREATE ROLE var_test_role; +CREATE VARIABLE public.v1 AS int DEFAULT 0; +-- check acl when variable is acessed by simple eval expr method +CREATE OR REPLACE FUNCTION public.fx_var(int) +RETURNS int AS $$ +DECLARE xx int; +BEGIN + xx := public.v1 + $1; + RETURN xx; +END; +$$ LANGUAGE plpgsql; +-- should be ok +SELECT public.fx_var(0); + fx_var +-------- + 0 +(1 row) + +SET ROLE TO var_test_role; +-- should to fail +SELECT public.fx_var(0); +ERROR: permission denied for session variable v1 +CONTEXT: PL/pgSQL function public.fx_var(integer) line 4 at assignment +SET ROLE TO default; +GRANT SELECT ON VARIABLE public.v1 TO var_test_role; +SET ROLE TO var_test_role; +-- should be ok +SELECT public.fx_var(0); + fx_var +-------- + 0 +(1 row) + +SET ROLE TO default; +REVOKE SELECT ON VARIABLE public.v1 FROM var_test_role; +SET ROLE TO var_test_role; +-- should be fail +SELECT public.fx_var(0); +ERROR: permission denied for session variable v1 +CONTEXT: PL/pgSQL function public.fx_var(integer) line 4 at assignment +SET ROLE TO DEFAULT; +DROP FUNCTION public.fx_var(int); +DROP VARIABLE public.v1; +DROP ROLE var_test_role; +CREATE TYPE public.svar_test_type AS (a int, b int, c numeric); +CREATE VARIABLE public.svar AS public.svar_test_type; +LET public.svar = ROW(10,20,30); +SELECT public.svar; + svar +------------ + (10,20,30) +(1 row) + +ALTER TYPE public.svar_test_type DROP ATTRIBUTE c; +-- should be ok +SELECT public.svar; + svar +--------- + (10,20) +(1 row) + +ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; +-- should be ok +SELECT public.svar; + svar +---------- + (10,20,) +(1 row) + +LET public.svar = ROW(10,20,30); +-- should be ok again for new value +SELECT public.svar; + svar +------------ + (10,20,30) +(1 row) + +DROP VARIABLE public.svar; +DROP TYPE public.svar_test_type; +CREATE VARIABLE public.svar AS int; +select oid as varid from pg_variable where varname = 'svar' and varnamespace = 'public'::regnamespace \gset +SELECT schema, name, removed FROM pg_session_variables(); + schema | name | removed +--------+------+--------- +(0 rows) + +LET public.svar = 100; +SELECT schema, name, removed FROM pg_session_variables(); + schema | name | removed +--------+------+--------- + public | svar | f +(1 row) + +BEGIN; +DROP VARIABLE public.svar; +-- value should be in memory +SELECT schema, removed FROM pg_session_variables(); + schema | removed +--------+--------- + | t +(1 row) + +ROLLBACK; +-- value should be in memory +SELECT schema, name, removed FROM pg_session_variables() f WHERE f.varid = :varid; + schema | name | removed +--------+------+--------- + public | svar | f +(1 row) + +SELECT public.svar; + svar +------ + 100 +(1 row) + +BEGIN; +DROP VARIABLE public.svar; +-- value should be in memory +SELECT schema, name = :varid::text AS ok, removed FROM pg_session_variables() f WHERE f.varid = :varid;; + schema | ok | removed +--------+----+--------- + | t | t +(1 row) + +COMMIT; +-- the memory should be clean; +SELECT schema, name, removed FROM pg_session_variables() WHERE f.varid = :varid; +ERROR: missing FROM-clause entry for table "f" +LINE 1: ..., name, removed FROM pg_session_variables() WHERE f.varid = ... + ^ +BEGIN; +CREATE VARIABLE public.svar AS int; +select oid as varid from pg_variable where varname = 'svar' and varnamespace = 'public'::regnamespace \gset +LET public.svar = 100; +ROLLBACK; +-- the memory should be clean; +SELECT schema, name, removed FROM pg_session_variables() f WHERE f.varid = :varid; + schema | name | removed +--------+------+--------- +(0 rows) + +\unset varid +CREATE VARIABLE public.svar AS int; +LET public.svar = 100; +-- repeated aborted transaction +BEGIN; DROP VARIABLE public.svar; ROLLBACK; +BEGIN; DROP VARIABLE public.svar; ROLLBACK; +BEGIN; DROP VARIABLE public.svar; ROLLBACK; +-- the value should be still available +SELECT public.svar; + svar +------ + 100 +(1 row) + +DROP VARIABLE public.svar; +CREATE TYPE public.svar_test_type AS (a int, b int); +CREATE VARIABLE public.svar AS public.svar_test_type; +SELECT public.svar; + svar +------ + +(1 row) + +ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; +SELECT public.svar; + svar +------ + +(1 row) + +ALTER TYPE public.svar_test_type DROP ATTRIBUTE b; +SELECT public.svar; + svar +------ + +(1 row) + +DROP VARIABLE public.svar; +DROP TYPE public.svar_test_type; +CREATE TYPE public.svar_test_type AS (a int, b int); +CREATE VARIABLE public.svar AS public.svar_test_type; +CREATE VARIABLE public.svar2 AS public.svar_test_type; +LET public.svar = (10, 20); +ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; +SELECT public.svar; + svar +---------- + (10,20,) +(1 row) + +LET public.svar2 = (10, 20, 30); +ALTER TYPE public.svar_test_type DROP ATTRIBUTE b; +SELECT public.svar; + svar +------- + (10,) +(1 row) + +SELECT public.svar2; + svar2 +--------- + (10,30) +(1 row) + +DROP VARIABLE public.svar; +DROP VARIABLE public.svar2; +DROP TYPE public.svar_test_type; +-- The composite type cannot be changed when it is used +CREATE TYPE public.svar_type AS (a int, b int); +CREATE VARIABLE public.svar AS public.svar_type; +-- should to fail +ALTER TYPE public.svar_type ALTER ATTRIBUTE b TYPE numeric; +ERROR: cannot alter type "svar_type" because session variable "public.svar" uses it +DROP VARIABLE public.svar; +CREATE TYPE public.svar_type2 AS (a int, b int, c public.svar_type); +CREATE VARIABLE public.svar AS public.svar_type2; +-- should to fail +ALTER TYPE public.svar_type ALTER ATTRIBUTE b TYPE numeric; +ERROR: cannot alter type "svar_type" because session variable "public.svar" uses it +DROP VARIABLE public.svar; +DROP TYPE public.svar_type2; +DROP TYPE public.svar_type; +-- The variable can be shadowed by table or by alias +CREATE TYPE public.svar_type AS (a int, b int, c int); +CREATE VARIABLE public.svar AS public.svar_type; +CREATE TABLE public.svar (a int, b int); +INSERT INTO public.svar VALUES(10, 20); +LET public.svar = (100, 200, 300); +-- should be ok +-- show table +SELECT * FROM public.svar; + a | b +----+---- + 10 | 20 +(1 row) + +SELECT svar.a FROM public.svar; + a +---- + 10 +(1 row) + +SELECT svar.* FROM public.svar; + a | b +----+---- + 10 | 20 +(1 row) + +-- show variable +SELECT public.svar; + svar +--------------- + (100,200,300) +(1 row) + +SELECT public.svar.c; + c +----- + 300 +(1 row) + +SELECT (public.svar).*; + a | b | c +-----+-----+----- + 100 | 200 | 300 +(1 row) + +-- the variable is shadowed, raise error +SELECT public.svar.c FROM public.svar; +ERROR: column svar.c does not exist +LINE 1: SELECT public.svar.c FROM public.svar; + ^ +-- can be fixed by alias +SELECT public.svar.c FROM public.svar x; + c +----- + 300 +(1 row) + +-- again with warnings +SET session_variables_ambiguity_warning TO on; +SELECT * FROM public.svar; + a | b +----+---- + 10 | 20 +(1 row) + +SELECT svar.a FROM public.svar; + a +---- + 10 +(1 row) + +SELECT svar.* FROM public.svar; + a | b +----+---- + 10 | 20 +(1 row) + +-- show variable +SELECT public.svar; + svar +--------------- + (100,200,300) +(1 row) + +SELECT public.svar.c; + c +----- + 300 +(1 row) + +SELECT (public.svar).*; + a | b | c +-----+-----+----- + 100 | 200 | 300 +(1 row) + +-- the variable is shadowed, raise error +SELECT public.svar.c FROM public.svar; +WARNING: session variable "public.svar" is shadowed +LINE 1: SELECT public.svar.c FROM public.svar; + ^ +DETAIL: Session variables can be shadowed by tables or table's aliases with the same name. +ERROR: column svar.c does not exist +LINE 1: SELECT public.svar.c FROM public.svar; + ^ +-- can be fixed by alias +SELECT public.svar.c FROM public.svar x; + c +----- + 300 +(1 row) + +SET session_variables_ambiguity_warning TO DEFAULT; +DROP VARIABLE public.svar; +DROP TABLE public.svar; +DROP TYPE public.svar_type; +-- the LET statement should be disallowed in CTE +CREATE VARIABLE public.svar AS int; +WITH x AS (LET public.svar = 100) SELECT * FROM x; +ERROR: LET not supported in WITH query +LINE 1: WITH x AS (LET public.svar = 100) SELECT * FROM x; + ^ +-- the LET statement can be prepared +PREPARE pp(int) AS LET public.svar = $1; +EXECUTE pp(10); +SELECT public.svar; + svar +------ + 10 +(1 row) + +EXECUTE pp(1000); +SELECT public.svar; + svar +------ + 1000 +(1 row) + +-- test of correct dependency +DROP VARIABLE public.svar; +EXECUTE pp(-1); +ERROR: session variable "public.svar" doesn't exist +DEALLOCATE pp; +CREATE VARIABLE public.svar2 AS int; +CREATE VARIABLE public.svar1 AS int DEFAULT 100; +PREPARE pp(int) AS LET public.svar2 = public.svar1 + $1; +EXECUTE pp(100); +SELECT public.svar1, public.svar2; + svar1 | svar2 +-------+------- + 100 | 200 +(1 row) + +LET public.svar1 = 1000; +EXECUTE pp(100); +SELECT public.svar1, public.svar2; + svar1 | svar2 +-------+------- + 1000 | 1100 +(1 row) + +DROP VARIABLE public.svar1; +EXECUTE pp(100); +ERROR: missing FROM-clause entry for table "public" +DEALLOCATE pp; +DROP VARIABLE public.svar2; +CREATE VARIABLE public.svar AS int; +-- should be ok +LET public.svar = generate_series(1, 1); +-- should fail +LET public.svar = generate_series(1, 2); +ERROR: expression returned more than one row +LET public.svar = generate_series(1, 0); +ERROR: expression returned no rows +DROP VARIABLE public.svar; +SET search_path TO DEFAULT; +CREATE TYPE ab AS (a integer, b integer); +CREATE VARIABLE v_ab AS ab; +CREATE TABLE v_ab (a integer, b integer); +SET session_variables_ambiguity_warning = 1; +-- warning should be raised +SELECT v_ab.a FROM v_ab; +WARNING: session variable "v_ab.a" is shadowed +LINE 1: SELECT v_ab.a FROM v_ab; + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. + a +--- +(0 rows) + +CREATE SCHEMA v_ab; +CREATE VARIABLE v_ab.a AS integer; +-- warning should be raised +SELECT v_ab.a FROM v_ab; +WARNING: session variable "v_ab.a" is shadowed +LINE 1: SELECT v_ab.a FROM v_ab; + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. + a +--- +(0 rows) + +DROP VARIABLE v_ab; +DROP TABLE v_ab; +DROP TYPE ab; +CREATE VARIABLE myvar AS int; +SELECT myvar.blabla; +ERROR: variable "public.myvar" is of type "integer", which is not a composite type +LINE 1: SELECT myvar.blabla; + ^ +DROP VARIABLE myvar; +-- the result of view should be same in parallel mode too +CREATE VARIABLE v1 AS int; +LET v1 = 10; +CREATE VIEW v1view AS SELECT COALESCE(v1, 0) AS result; +SELECT * FROM v1view; + result +-------- + 10 +(1 row) + +SET debug_parallel_query TO on; +SELECT * FROM v1view; + result +-------- + 10 +(1 row) + +SET debug_parallel_query TO off; +DROP VIEW v1view; +DROP VARIABLE v1; +-- the value should not be corrupted +CREATE VARIABLE v1 text DEFAULT 'abc'; +CREATE FUNCTION ffunc() +RETURNS text AS $$ +BEGIN + RETURN gfunc(v1); +END +$$ LANGUAGE plpgsql; +CREATE FUNCTION gfunc(t text) +RETURNS text AS $$ +BEGIN + LET v1 = 'BOOM!'; + RETURN t; +END; +$$ LANGUAGE plpgsql; +select ffunc(); + ffunc +------- + abc +(1 row) + +DROP FUNCTION ffunc(); +DROP FUNCTION gfunc(text); +DROP VARIABLE v1; +CREATE TYPE t_am_type AS (b int); +CREATE SCHEMA xxx_am; +SET search_path TO public; +CREATE VARIABLE xxx_am AS t_am_type DEFAULT row(10); +SELECT xxx_am; + xxx_am +-------- + (10) +(1 row) + +SELECT public.xxx_am; + xxx_am +-------- + (10) +(1 row) + +CREATE VARIABLE xxx_am.b AS int DEFAULT 20; +-- the reference should be ambiguous +SELECT xxx_am.b; +ERROR: session variable reference "xxx_am.b" is ambiguous +LINE 1: SELECT xxx_am.b; + ^ +-- enhanced references should be ok +SELECT public.xxx_am.b; + b +---- + 10 +(1 row) + +SELECT :"DBNAME".xxx_am.b; + b +---- + 20 +(1 row) + +SET session_variables_ambiguity_warning TO on; +CREATE TABLE xxx_am(b int); +-- should be warning, not error +SELECT xxx_am.b FROM xxx_am; +WARNING: session variable "xxx_am.b" is shadowed +LINE 1: SELECT xxx_am.b FROM xxx_am; + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. + b +--- +(0 rows) + +-- no warning +SELECT x.b FROM xxx_am x; + b +--- +(0 rows) + +DROP TABLE xxx_am; +DROP VARIABLE public.xxx_am; +DROP VARIABLE xxx_am.b; +DROP SCHEMA xxx_am; +CREATE SCHEMA :"DBNAME"; +CREATE VARIABLE :"DBNAME".:"DBNAME".:"DBNAME" AS t_am_type DEFAULT row(10); +CREATE VARIABLE :"DBNAME".:"DBNAME".b AS int DEFAULT 20; +SET search_path to :"DBNAME"; +-- In this test case, error (and warning) messages contains database name. +-- It is "regression", but it can be possibly different, so we try to +-- supress error and warning messages. +\set ECHO none +ERROR: session variable reference "regression.b" is ambiguous +LINE 1: SELECT "regression".b; + ^ +true, 42P08 +ERROR: session variable reference "regression.regression.b" is ambiguous +LINE 1: SELECT "regression"."regression".b; + ^ +true, 42P08 + b +--- +(0 rows) + +false, 00000 +DROP TABLE :"DBNAME"; +DROP VARIABLE :"DBNAME".:"DBNAME".b; +DROP VARIABLE :"DBNAME".:"DBNAME".:"DBNAME"; +DROP SCHEMA :"DBNAME"; +SET client_min_messages TO DEFAULT; +SET search_path to public; diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index cf46fa3359..a45cd40a66 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -111,7 +111,7 @@ test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson # NB: temp.sql does a reconnect which transiently uses 2 connections, # so keep this parallel group to at most 19 tests # ---------- -test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion truncate alter_table sequence polymorphism rowtypes returning largeobject with xml +test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion truncate alter_table sequence polymorphism rowtypes returning largeobject with xml session_variables # ---------- # Another group of parallel tests diff --git a/src/test/regress/sql/session_variables.sql b/src/test/regress/sql/session_variables.sql new file mode 100644 index 0000000000..42926dfbcc --- /dev/null +++ b/src/test/regress/sql/session_variables.sql @@ -0,0 +1,1203 @@ +-- should fail, pseudotypes are not allowed +CREATE VARIABLE xx AS anyelement; + +-- should be ok +CREATE SCHEMA svartest CREATE VARIABLE xx AS int CREATE TABLE foo(a int); + +LET svartest.xx = 100; + +DROP SCHEMA svartest CASCADE; + +-- should fail +LET svartest.xx = 100; + +CREATE SCHEMA svartest; + +SET search_path = svartest; + +CREATE VARIABLE var1 AS integer; +CREATE TEMP VARIABLE var2 AS text; + +DROP VARIABLE var1, var2; + +-- functional interface +CREATE VARIABLE var1 AS numeric; + +CREATE ROLE var_test_role; +GRANT USAGE ON SCHEMA svartest TO var_test_role; + +SET ROLE TO var_test_role; + +-- should fail +SELECT var1; + +SET ROLE TO DEFAULT; + +GRANT SELECT ON VARIABLE var1 TO var_test_role; + +SET ROLE TO var_test_role; +-- should fail +LET var1 = 10; +-- should work +SELECT var1; + +SET ROLE TO DEFAULT; + +GRANT UPDATE ON VARIABLE var1 TO var_test_role; + +SET ROLE TO var_test_role; + +-- should work +LET var1 = 333; + +SET ROLE TO DEFAULT; + +REVOKE ALL ON VARIABLE var1 FROM var_test_role; + +CREATE OR REPLACE FUNCTION secure_var() +RETURNS int AS $$ + SELECT svartest.var1::int; +$$ LANGUAGE sql SECURITY DEFINER; + +SELECT secure_var(); + +SET ROLE TO var_test_role; + +-- should fail +SELECT svartest.var1; + +-- should work; +SELECT secure_var(); + +SET ROLE TO DEFAULT; + +EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM generate_series(1,100) g(v) WHERE v = var1; + +CREATE VIEW schema_var_view AS SELECT var1; + +SELECT * FROM schema_var_view; + +\c - + +SET search_path = svartest; + +-- should work still, but var will be empty +SELECT * FROM schema_var_view; + +LET var1 = pi(); + +SELECT var1; + +-- we can see execution plan of LET statement +EXPLAIN (VERBOSE, COSTS OFF) LET var1 = pi(); + +SELECT var1; + +CREATE VARIABLE var3 AS int; + +CREATE OR REPLACE FUNCTION inc(int) +RETURNS int AS $$ +BEGIN + LET svartest.var3 = COALESCE(svartest.var3 + $1, $1); + RETURN var3; +END; +$$ LANGUAGE plpgsql; + +SELECT inc(1); +SELECT inc(1); +SELECT inc(1); + +SELECT inc(1) FROM generate_series(1,10); + +SET ROLE TO var_test_role; + +-- should fail +LET var3 = 0; + +SET ROLE TO DEFAULT; + +DROP VIEW schema_var_view; + +DROP VARIABLE var1 CASCADE; +DROP VARIABLE var3 CASCADE; + +-- composite variables + +CREATE TYPE sv_xyz AS (x int, y int, z numeric(10,2)); + +CREATE VARIABLE v1 AS sv_xyz; +CREATE VARIABLE v2 AS sv_xyz; + +\d v1 +\d v2 + +LET v1 = (1,2,3.14); +LET v2 = (10,20,3.14*10); + +-- should work too - there are prepared casts +LET v1 = (1,2,3.14); + +SELECT v1; +SELECT v2; +SELECT (v1).*; +SELECT (v2).*; + +SELECT v1.x + v1.z; +SELECT v2.x + v2.z; + +-- access to composite fields should be safe too +-- should fail +SET ROLE TO var_test_role; + +SELECT v2.x; + +SET ROLE TO DEFAULT; + +DROP VARIABLE v1; +DROP VARIABLE v2; + +REVOKE USAGE ON SCHEMA svartest FROM var_test_role; +DROP ROLE var_test_role; + +-- scalar variables should not be in conflict with qualified column +CREATE VARIABLE varx AS text; +SELECT varx.relname FROM pg_class varx WHERE varx.relname = 'pg_class'; + +-- should fail +SELECT varx.xxx; + +-- don't allow multi column query +CREATE TYPE vartesttp AS (a1 int, b1 int, c1 int); +CREATE VARIABLE v1 AS vartesttp; + +-- should fail +LET v1 = (NULL::vartesttp).*; + +DROP VARIABLE v1; +DROP TYPE vartesttp; + +-- variables can be updated under RO transaction + +BEGIN; +SET TRANSACTION READ ONLY; +LET varx = 'hello'; +COMMIT; + +SELECT varx; + +DROP VARIABLE varx; + +CREATE TYPE t1 AS (a int, b numeric, c text); + +CREATE VARIABLE v1 AS t1; +LET v1 = (1, pi(), 'hello'); +SELECT v1; +LET v1.b = 10.2222; +SELECT v1; + +-- should fail +LET v1.x = 10; + +DROP VARIABLE v1; +DROP TYPE t1; + +-- arrays are supported +CREATE VARIABLE va1 AS numeric[]; +LET va1 = ARRAY[1.1,2.1]; +LET va1[1] = 10.1; +SELECT va1; + +-- LET target doesn't allow srf, should fail +LET va1[generate_series(1,3)] = 100; + +CREATE TYPE ta2 AS (a numeric, b numeric[]); +CREATE VARIABLE va2 AS ta2; +LET va2 = (10.1, ARRAY[0.0, 0.0]); +LET va2.a = 10.2; +SELECT va2; +LET va2.b[1] = 10.3; +SELECT va2; + +DROP VARIABLE va1; +DROP VARIABLE va2; +DROP TYPE ta2; + +-- default values +CREATE VARIABLE v1 AS numeric DEFAULT pi(); +LET v1 = v1 * 2; +SELECT v1; + +CREATE TYPE t2 AS (a numeric, b text); +CREATE VARIABLE v2 AS t2 DEFAULT (NULL, 'Hello'); +LET svartest.v2.a = pi(); +SELECT v2; + +-- should fail due dependency +DROP TYPE t2; + +-- should be ok +DROP VARIABLE v1; +DROP VARIABLE v2; + +-- tests of alters +CREATE SCHEMA var_schema1; +CREATE SCHEMA var_schema2; + +CREATE VARIABLE var_schema1.var1 AS integer; +LET var_schema1.var1 = 1000; +SELECT var_schema1.var1; +ALTER VARIABLE var_schema1.var1 SET SCHEMA var_schema2; +SELECT var_schema2.var1; + +CREATE ROLE var_test_role; + +ALTER VARIABLE var_schema2.var1 OWNER TO var_test_role; +SET ROLE TO var_test_role; + +-- should fail, no access to schema var_schema2.var +SELECT var_schema2.var1; +DROP VARIABLE var_schema2.var1; + +SET ROLE TO DEFAULT; + +ALTER VARIABLE var_schema2.var1 SET SCHEMA public; + +SET ROLE TO var_test_role; +SELECT public.var1; + +ALTER VARIABLE public.var1 RENAME TO var1_renamed; + +SELECT public.var1_renamed; + +DROP VARIABLE public.var1_renamed; + +SET ROLE TO DEFAULt; + +-- default rights test +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON VARIABLES TO var_test_role; + +CREATE VARIABLE public.var2 AS int; + +SET ROLE TO var_test_role; + +-- should be ok +LET public.var2 = 100; +SELECT public.var2; + +SET ROLE TO DEFAULt; + +DROP VARIABLE public.var2; +DROP OWNED BY var_test_role; + +DROP ROLE var_test_role; + +CREATE VARIABLE xx AS text DEFAULT 'hello'; + +SELECT xx, upper(xx); + +LET xx = 'Hi'; + +SELECT xx; + +DROP VARIABLE xx; + +-- ON TRANSACTION END RESET tests +CREATE VARIABLE t1 AS int DEFAULT -1 ON TRANSACTION END RESET; + +BEGIN; + SELECT t1; + LET t1 = 100; + SELECT t1; +COMMIT; + +SELECT t1; + +BEGIN; + SELECT t1; + LET t1 = 100; + SELECT t1; +ROLLBACK; + +SELECT t1; + +DROP VARIABLE t1; + +CREATE VARIABLE v1 AS int DEFAULT 0; +CREATE VARIABLE v2 AS text DEFAULT 'none'; + +LET v1 = 100; +LET v2 = 'Hello'; +SELECT v1, v2; +LET v1 = DEFAULT; +LET v2 = DEFAULT; +SELECT v1, v2; + +DROP VARIABLE v1; +DROP VARIABLE v2; + +-- ON COMMIT DROP tests +-- should be 0 always +SELECT count(*) FROM pg_variable; + +CREATE TEMP VARIABLE g AS int ON COMMIT DROP; + +SELECT count(*) FROM pg_variable; + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; +COMMIT; + +SELECT count(*) FROM pg_variable; + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; +ROLLBACK; + +SELECT count(*) FROM pg_variable; + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; + LET g = 1; + DISCARD VARIABLES; +COMMIT; + +SELECT count(*) FROM pg_variable; + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; + LET g = 1; + DISCARD VARIABLES; +ROLLBACK; + +SELECT count(*) FROM pg_variable; + +-- Encourage use of parallel plans +SET parallel_setup_cost = 0; +SET parallel_tuple_cost = 0; +SET min_parallel_table_scan_size = 0; +SET max_parallel_workers_per_gather = 2; + +-- test on query with workers +CREATE TABLE svar_test(a int); +INSERT INTO svar_test SELECT * FROM generate_series(1,1000); +ANALYZE svar_test; +CREATE VARIABLE zero int; +LET zero = 0; + +-- parallel workers should be used +EXPLAIN (costs off) SELECT count(*) FROM svar_test WHERE a%10 = zero; + +-- result should be 100 +SELECT count(*) FROM svar_test WHERE a%10 = zero; + +LET zero = (SELECT count(*) FROM svar_test); + +-- result should be 1000 +SELECT zero; + +-- parallel workers should be used +EXPLAIN (costs off) LET zero = (SELECT count(*) FROM svar_test); + +-- test for dependency on relation +CREATE VARIABLE v_table AS svar_test; + +ALTER TABLE svar_test ALTER COLUMN a TYPE text; + +DROP VARIABLE v_table; +DROP TABLE svar_test; +DROP VARIABLE zero; + +RESET parallel_setup_cost; +RESET parallel_tuple_cost; +RESET min_parallel_table_scan_size; +RESET max_parallel_workers_per_gather; + +-- use variables in prepared statements +CREATE VARIABLE v AS numeric; +LET v = 3.14; + +-- use variables in views +CREATE VIEW vv AS SELECT COALESCE(v, 0) + 1000 AS result; +SELECT * FROM vv; + +-- start a new session +\c + +SET search_path to svartest; + +SELECT * FROM vv; +LET v = 3.14; +SELECT * FROM vv; + +-- should fail, dependency +DROP VARIABLE v; + +-- should be ok +DROP VARIABLE v CASCADE; + +-- other features +CREATE VARIABLE dt AS integer DEFAULT 0; + +LET dt = 100; +SELECT dt; + +DISCARD VARIABLES; + +SELECT dt; + +DROP VARIABLE dt; + +-- NOT NULL +CREATE VARIABLE v1 AS int NOT NULL; +CREATE VARIABLE v2 AS int NOT NULL DEFAULT NULL; + +-- should fail +SELECT v1; +SELECT v2; +LET v1 = NULL; +LET v2 = NULL; +LET v1 = DEFAULT; +LET v2 = DEFAULT; + +-- should be ok +LET v1 = 100; +LET v2 = 1000; +SELECT v1, v2; + +DROP VARIABLE v1; +DROP VARIABLE v2; + +CREATE VARIABLE tv AS int; +CREATE VARIABLE IF NOT EXISTS tv AS int; +DROP VARIABLE tv; + +CREATE IMMUTABLE VARIABLE iv AS int DEFAULT 100; +SELECT iv; + +-- should fail; +LET iv = 10000; + +DROP VARIABLE iv; + +-- different order +CREATE IMMUTABLE VARIABLE iv AS int DEFAULT 100; +-- should to fail +LET iv = 10000; +-- should be ok +SELECT iv; + +DROP VARIABLE iv; + +CREATE IMMUTABLE VARIABLE iv AS int; + +-- should be ok +LET iv = NULL; + +-- should fail +LET iv = NULL; + +DROP VARIABLE iv; + +-- create variable inside plpgsql block +DO $$ +BEGIN + CREATE VARIABLE do_test_svar AS date DEFAULT '2000-01-01'; +END; +$$; + +SELECT do_test_svar; + +DROP VARIABLE do_test_svar; + +-- should fail +CREATE IMMUTABLE VARIABLE xx AS int NOT NULL; + + + +-- REASSIGN OWNED test +CREATE ROLE var_test_role1; +CREATE ROLE var_test_role2; + +CREATE VARIABLE xxx_var AS int; + +ALTER VARIABLE xxx_var OWNER TO var_test_role1; +REASSIGN OWNED BY var_test_role1 to var_test_role2; + +SELECT varowner::regrole FROM pg_variable WHERE varname = 'xxx_var'; + +DROP OWNED BY var_test_role1; +DROP ROLE var_test_role1; +SELECT count(*) FROM pg_variable WHERE varname = 'xxx_var'; + +DROP OWNED BY var_test_role2; +DROP ROLE var_test_role2; +SELECT count(*) FROM pg_variable WHERE varname = 'xxx_var'; + +-- creating, dropping temporary variable +BEGIN; + +CREATE TEMP VARIABLE tempvar AS INT ON COMMIT DROP; + +LET tempvar = 100; + +SAVEPOINT s1; + +DROP VARIABLE tempvar; + +ROLLBACK TO s1; + +SELECT tempvar; + +COMMIT; + +-- should to fail +LET tempvar = 100; + +BEGIN; + +SAVEPOINT s1; + +CREATE TEMP VARIABLE tempvar AS INT ON COMMIT DROP; + +LET tempvar = 100; + +ROLLBACK TO s1; + +COMMIT; + +-- should to fail +LET tempvar = 100; + +CREATE VARIABLE var1 AS int; +LET var1 = 100; +BEGIN; +DROP VARIABLE var1; +ROLLBACK; +SELECT var1; + +DROP VARIABLE var1; + +CREATE VARIABLE var1 AS int DEFAULT 100; +COMMENT ON VARIABLE var1 IS 'some variable comment'; + +SELECT pg_catalog.obj_description(oid, 'pg_variable') FROM pg_variable WHERE varname = 'var1'; + +DROP VARIABLE var1; + +CREATE TABLE xxtab(avar int); + +CREATE TYPE xxtype AS (avar int); + +CREATE VARIABLE xxtab AS xxtype; + +INSERT INTO xxtab VALUES(10); + +-- it is ambiguous, but columns are preferred +-- variable is shadowed +SELECT xxtab.avar FROM xxtab; + +SET session_variables_ambiguity_warning TO on; + +SELECT xxtab.avar FROM xxtab; + +SET search_path = svartest; + +CREATE VARIABLE testvar as int; + +-- plpgsql variables are preferred against session variables +DO $$ +<<myblock>> +DECLARE testvar int; +BEGIN + -- should be ok without warning + LET testvar = 100; + -- should be ok without warning + testvar := 1000; + -- should be ok without warning + RAISE NOTICE 'session variable is %', svartest.testvar; + -- should be ok without warning + RAISE NOTICE 'plpgsql variable is %', myblock.testvar; + -- should to print plpgsql variable with warning + RAISE NOTICE 'variable is %', testvar; +END; +$$; + +DROP VARIABLE testvar; + +SET session_variables_ambiguity_warning TO default; + +-- should be ok +SELECT avar FROM xxtab; + +CREATE VARIABLE public.avar AS int; + +-- should be ok, see the table +SELECT avar FROM xxtab; + +-- should be ok +SELECT public.avar FROM xxtab; + +DROP VARIABLE xxtab; + +SELECT xxtab.avar FROM xxtab; + +DROP VARIABLE public.avar; + +DROP TYPE xxtype; + +DROP TABLE xxtab; + +-- test of plan cache invalidation +CREATE VARIABLE xx AS int; + +SET plan_cache_mode = force_generic_plan; + +PREPARE pp AS SELECT xx; + +EXECUTE pp; + +DROP VARIABLE xx; + +CREATE VARIABLE xx AS int; + +-- should to work +EXECUTE pp; + +DROP VARIABLE xx; + +DEALLOCATE pp; + +SET plan_cache_mode = DEFAULT; + +CREATE ROLE var_test_role; + +CREATE SCHEMA vartest; + +GRANT USAGE ON SCHEMA vartest TO var_test_role; + +CREATE VARIABLE vartest.x AS int; +CREATE VARIABLE vartest.y AS int; + +LET vartest.x = 100; +LET vartest.y = 101; + +GRANT SELECT ON ALL VARIABLES IN SCHEMA vartest TO var_test_role; + +SET ROLE TO var_test_role; + +SELECT vartest.x, vartest.y; + +SET ROLE TO DEFAULT; + +REVOKE SELECT ON ALL VARIABLES IN SCHEMA vartest FROM var_test_role; + +SET ROLE TO var_test_role; + +-- should to fail +SELECT vartest.x; +SELECT vartest.y; + +SET ROLE TO DEFAULT; + +DROP VARIABLE vartest.x, vartest.y; + +DROP SCHEMA vartest; + +DROP ROLE var_test_role; + +-- test cached plan +CREATE VARIABLE v1 AS text; +CREATE VARIABLE v2 AS int; +CREATE VARIABLE v3 AS int; + +LET v1 = 'test'; +LET v2 = 10; +LET v3 = 5; + +PREPARE q1 AS SELECT v1 || i FROM generate_series(1, v2) g(i) WHERE i IN (v2, v3); + +SET plan_cache_mode to force_generic_plan; + +EXECUTE q1; + +EXPLAIN EXECUTE q1; + +-- dependecy check +DROP VARIABLE v3; + +-- recreate v3 again +CREATE VARIABLE v3 AS int DEFAULT 6; + +-- should to work, the plan should be recreated +EXECUTE q1; + +DEALLOCATE q1; + +-- fill v1 by long text +LET v1 = repeat(' ', 10000); + +PREPARE q1 AS SELECT length(v1); + +EXECUTE q1; + +LET v1 = repeat(' ', 5000); + +EXECUTE q1; + +DEALLOCATE q1; + +SET plan_cache_mode to default; + +DROP VARIABLE v1, v2, v3; + +CREATE ROLE var_test_role; + +CREATE VARIABLE public.v1 AS int DEFAULT 0; + +-- check acl when variable is acessed by simple eval expr method +CREATE OR REPLACE FUNCTION public.fx_var(int) +RETURNS int AS $$ +DECLARE xx int; +BEGIN + xx := public.v1 + $1; + RETURN xx; +END; +$$ LANGUAGE plpgsql; + +-- should be ok +SELECT public.fx_var(0); + +SET ROLE TO var_test_role; + +-- should to fail +SELECT public.fx_var(0); + +SET ROLE TO default; + +GRANT SELECT ON VARIABLE public.v1 TO var_test_role; + +SET ROLE TO var_test_role; + +-- should be ok +SELECT public.fx_var(0); + +SET ROLE TO default; + +REVOKE SELECT ON VARIABLE public.v1 FROM var_test_role; + +SET ROLE TO var_test_role; + +-- should be fail +SELECT public.fx_var(0); + +SET ROLE TO DEFAULT; + +DROP FUNCTION public.fx_var(int); + +DROP VARIABLE public.v1; + +DROP ROLE var_test_role; + +CREATE TYPE public.svar_test_type AS (a int, b int, c numeric); + +CREATE VARIABLE public.svar AS public.svar_test_type; + +LET public.svar = ROW(10,20,30); + +SELECT public.svar; + +ALTER TYPE public.svar_test_type DROP ATTRIBUTE c; + +-- should be ok +SELECT public.svar; + +ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; + +-- should be ok +SELECT public.svar; + +LET public.svar = ROW(10,20,30); + +-- should be ok again for new value +SELECT public.svar; + +DROP VARIABLE public.svar; + +DROP TYPE public.svar_test_type; + +CREATE VARIABLE public.svar AS int; +select oid as varid from pg_variable where varname = 'svar' and varnamespace = 'public'::regnamespace \gset + +SELECT schema, name, removed FROM pg_session_variables(); + +LET public.svar = 100; + +SELECT schema, name, removed FROM pg_session_variables(); + +BEGIN; + +DROP VARIABLE public.svar; + +-- value should be in memory +SELECT schema, removed FROM pg_session_variables(); + +ROLLBACK; + +-- value should be in memory +SELECT schema, name, removed FROM pg_session_variables() f WHERE f.varid = :varid; + +SELECT public.svar; + +BEGIN; + +DROP VARIABLE public.svar; + +-- value should be in memory +SELECT schema, name = :varid::text AS ok, removed FROM pg_session_variables() f WHERE f.varid = :varid;; + +COMMIT; + +-- the memory should be clean; +SELECT schema, name, removed FROM pg_session_variables() WHERE f.varid = :varid; + +BEGIN; + +CREATE VARIABLE public.svar AS int; +select oid as varid from pg_variable where varname = 'svar' and varnamespace = 'public'::regnamespace \gset + + +LET public.svar = 100; + +ROLLBACK; + +-- the memory should be clean; +SELECT schema, name, removed FROM pg_session_variables() f WHERE f.varid = :varid; + +\unset varid + +CREATE VARIABLE public.svar AS int; + +LET public.svar = 100; + +-- repeated aborted transaction +BEGIN; DROP VARIABLE public.svar; ROLLBACK; +BEGIN; DROP VARIABLE public.svar; ROLLBACK; +BEGIN; DROP VARIABLE public.svar; ROLLBACK; + +-- the value should be still available +SELECT public.svar; + +DROP VARIABLE public.svar; + +CREATE TYPE public.svar_test_type AS (a int, b int); + +CREATE VARIABLE public.svar AS public.svar_test_type; + +SELECT public.svar; + +ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; + +SELECT public.svar; + +ALTER TYPE public.svar_test_type DROP ATTRIBUTE b; + +SELECT public.svar; + +DROP VARIABLE public.svar; +DROP TYPE public.svar_test_type; + +CREATE TYPE public.svar_test_type AS (a int, b int); + +CREATE VARIABLE public.svar AS public.svar_test_type; +CREATE VARIABLE public.svar2 AS public.svar_test_type; + +LET public.svar = (10, 20); + +ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; + +SELECT public.svar; + +LET public.svar2 = (10, 20, 30); + +ALTER TYPE public.svar_test_type DROP ATTRIBUTE b; + +SELECT public.svar; +SELECT public.svar2; + +DROP VARIABLE public.svar; +DROP VARIABLE public.svar2; +DROP TYPE public.svar_test_type; + +-- The composite type cannot be changed when it is used +CREATE TYPE public.svar_type AS (a int, b int); + +CREATE VARIABLE public.svar AS public.svar_type; + +-- should to fail +ALTER TYPE public.svar_type ALTER ATTRIBUTE b TYPE numeric; + +DROP VARIABLE public.svar; + +CREATE TYPE public.svar_type2 AS (a int, b int, c public.svar_type); + +CREATE VARIABLE public.svar AS public.svar_type2; + +-- should to fail +ALTER TYPE public.svar_type ALTER ATTRIBUTE b TYPE numeric; + +DROP VARIABLE public.svar; +DROP TYPE public.svar_type2; +DROP TYPE public.svar_type; + +-- The variable can be shadowed by table or by alias +CREATE TYPE public.svar_type AS (a int, b int, c int); +CREATE VARIABLE public.svar AS public.svar_type; + +CREATE TABLE public.svar (a int, b int); + +INSERT INTO public.svar VALUES(10, 20); + +LET public.svar = (100, 200, 300); + +-- should be ok +-- show table +SELECT * FROM public.svar; +SELECT svar.a FROM public.svar; +SELECT svar.* FROM public.svar; + +-- show variable +SELECT public.svar; +SELECT public.svar.c; +SELECT (public.svar).*; + +-- the variable is shadowed, raise error +SELECT public.svar.c FROM public.svar; + +-- can be fixed by alias +SELECT public.svar.c FROM public.svar x; + +-- again with warnings +SET session_variables_ambiguity_warning TO on; + +SELECT * FROM public.svar; +SELECT svar.a FROM public.svar; +SELECT svar.* FROM public.svar; + +-- show variable +SELECT public.svar; +SELECT public.svar.c; +SELECT (public.svar).*; + +-- the variable is shadowed, raise error +SELECT public.svar.c FROM public.svar; + +-- can be fixed by alias +SELECT public.svar.c FROM public.svar x; + +SET session_variables_ambiguity_warning TO DEFAULT; + +DROP VARIABLE public.svar; +DROP TABLE public.svar; +DROP TYPE public.svar_type; + +-- the LET statement should be disallowed in CTE +CREATE VARIABLE public.svar AS int; +WITH x AS (LET public.svar = 100) SELECT * FROM x; + +-- the LET statement can be prepared +PREPARE pp(int) AS LET public.svar = $1; +EXECUTE pp(10); +SELECT public.svar; +EXECUTE pp(1000); +SELECT public.svar; + +-- test of correct dependency +DROP VARIABLE public.svar; +EXECUTE pp(-1); + +DEALLOCATE pp; + +CREATE VARIABLE public.svar2 AS int; +CREATE VARIABLE public.svar1 AS int DEFAULT 100; + +PREPARE pp(int) AS LET public.svar2 = public.svar1 + $1; + +EXECUTE pp(100); + +SELECT public.svar1, public.svar2; + +LET public.svar1 = 1000; + +EXECUTE pp(100); + +SELECT public.svar1, public.svar2; + +DROP VARIABLE public.svar1; + +EXECUTE pp(100); + +DEALLOCATE pp; + +DROP VARIABLE public.svar2; + +CREATE VARIABLE public.svar AS int; + +-- should be ok +LET public.svar = generate_series(1, 1); + +-- should fail +LET public.svar = generate_series(1, 2); +LET public.svar = generate_series(1, 0); + +DROP VARIABLE public.svar; + +SET search_path TO DEFAULT; + +CREATE TYPE ab AS (a integer, b integer); + +CREATE VARIABLE v_ab AS ab; + +CREATE TABLE v_ab (a integer, b integer); + +SET session_variables_ambiguity_warning = 1; + +-- warning should be raised +SELECT v_ab.a FROM v_ab; + +CREATE SCHEMA v_ab; + +CREATE VARIABLE v_ab.a AS integer; + +-- warning should be raised +SELECT v_ab.a FROM v_ab; + +DROP VARIABLE v_ab; +DROP TABLE v_ab; +DROP TYPE ab; + +CREATE VARIABLE myvar AS int; + +SELECT myvar.blabla; + +DROP VARIABLE myvar; + +-- the result of view should be same in parallel mode too +CREATE VARIABLE v1 AS int; +LET v1 = 10; + +CREATE VIEW v1view AS SELECT COALESCE(v1, 0) AS result; + +SELECT * FROM v1view; + +SET debug_parallel_query TO on; + +SELECT * FROM v1view; + +SET debug_parallel_query TO off; + +DROP VIEW v1view; +DROP VARIABLE v1; + +-- the value should not be corrupted +CREATE VARIABLE v1 text DEFAULT 'abc'; + +CREATE FUNCTION ffunc() +RETURNS text AS $$ +BEGIN + RETURN gfunc(v1); +END +$$ LANGUAGE plpgsql; + +CREATE FUNCTION gfunc(t text) +RETURNS text AS $$ +BEGIN + LET v1 = 'BOOM!'; + RETURN t; +END; +$$ LANGUAGE plpgsql; + +select ffunc(); + +DROP FUNCTION ffunc(); +DROP FUNCTION gfunc(text); + +DROP VARIABLE v1; + +CREATE TYPE t_am_type AS (b int); +CREATE SCHEMA xxx_am; + +SET search_path TO public; + +CREATE VARIABLE xxx_am AS t_am_type DEFAULT row(10); +SELECT xxx_am; +SELECT public.xxx_am; + +CREATE VARIABLE xxx_am.b AS int DEFAULT 20; + +-- the reference should be ambiguous +SELECT xxx_am.b; + +-- enhanced references should be ok +SELECT public.xxx_am.b; +SELECT :"DBNAME".xxx_am.b; + +SET session_variables_ambiguity_warning TO on; + +CREATE TABLE xxx_am(b int); + +-- should be warning, not error +SELECT xxx_am.b FROM xxx_am; + +-- no warning +SELECT x.b FROM xxx_am x; + +DROP TABLE xxx_am; +DROP VARIABLE public.xxx_am; +DROP VARIABLE xxx_am.b; +DROP SCHEMA xxx_am; + +CREATE SCHEMA :"DBNAME"; + +CREATE VARIABLE :"DBNAME".:"DBNAME".:"DBNAME" AS t_am_type DEFAULT row(10); +CREATE VARIABLE :"DBNAME".:"DBNAME".b AS int DEFAULT 20; + +SET search_path to :"DBNAME"; + +-- In this test case, error (and warning) messages contains database name. +-- It is "regression", but it can be possibly different, so we try to +-- supress error and warning messages. + +\set ECHO none +SET client_min_messages TO error; + +-- should be ambiguous +SELECT :"DBNAME".b; + +-- should be true +\echo :ERROR, :SQLSTATE + +-- should be ambiguous too +SELECT :"DBNAME".:"DBNAME".b; + +-- should be true +\echo :ERROR, :SQLSTATE + +CREATE TABLE :"DBNAME"(b int); + +-- should be warning, not error +SELECT :"DBNAME".b FROM :"DBNAME"; + +-- should be false +\echo :ERROR, :SQLSTATE + +\set ECHO all + +DROP TABLE :"DBNAME"; + +DROP VARIABLE :"DBNAME".:"DBNAME".b; +DROP VARIABLE :"DBNAME".:"DBNAME".:"DBNAME"; +DROP SCHEMA :"DBNAME"; + +SET client_min_messages TO DEFAULT; + +SET search_path to public; -- 2.40.1 [text/x-patch] v20230516-1-0010-documentation.patch (46.1K, ../../CAFj8pRDF_eygHWcG8Y9d73zmus=ohVup-ngP7w-m4tbtWg14KQ@mail.gmail.com/7-v20230516-1-0010-documentation.patch) download | inline diff: From 5141632a10cb2508fa06131afc0e514bff30443a Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Mon, 4 Apr 2022 20:23:54 +0200 Subject: [PATCH 10/10] documentation Documentation for CREATE VARIABLE, DROP VARIABLE and LET commands. Update of GRANT, REVOKE, DISCARD, ALTER commands related to support of session variables. --- doc/src/sgml/catalogs.sgml | 170 +++++++++++++ doc/src/sgml/config.sgml | 60 +++++ doc/src/sgml/ddl.sgml | 64 +++++ doc/src/sgml/event-trigger.sgml | 24 ++ doc/src/sgml/glossary.sgml | 16 ++ doc/src/sgml/plpgsql.sgml | 12 + doc/src/sgml/ref/allfiles.sgml | 4 + .../sgml/ref/alter_default_privileges.sgml | 26 +- doc/src/sgml/ref/alter_variable.sgml | 179 ++++++++++++++ doc/src/sgml/ref/comment.sgml | 1 + doc/src/sgml/ref/create_schema.sgml | 7 +- doc/src/sgml/ref/create_variable.sgml | 223 ++++++++++++++++++ doc/src/sgml/ref/discard.sgml | 14 +- doc/src/sgml/ref/drop_variable.sgml | 118 +++++++++ doc/src/sgml/ref/grant.sgml | 6 + doc/src/sgml/ref/let.sgml | 109 +++++++++ doc/src/sgml/ref/pg_restore.sgml | 11 + doc/src/sgml/ref/revoke.sgml | 7 + doc/src/sgml/reference.sgml | 4 + 19 files changed, 1043 insertions(+), 12 deletions(-) create mode 100644 doc/src/sgml/ref/alter_variable.sgml create mode 100644 doc/src/sgml/ref/create_variable.sgml create mode 100644 doc/src/sgml/ref/drop_variable.sgml create mode 100644 doc/src/sgml/ref/let.sgml diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 5240840552..45989060f2 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -369,6 +369,11 @@ <entry><link linkend="catalog-pg-user-mapping"><structname>pg_user_mapping</structname></link></entry> <entry>mappings of users to foreign servers</entry> </row> + + <row> + <entry><link linkend="catalog-pg-variable"><structname>pg_variable</structname></link></entry> + <entry>session variables</entry> + </row> </tbody> </tgroup> </table> @@ -9677,4 +9682,169 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </table> </sect1> + <sect1 id="catalog-pg-variable"> + <title><structname>pg_variable</structname></title> + + <indexterm zone="catalog-pg-variable"> + <primary>pg_variable</primary> + </indexterm> + + <para> + The table <structname>pg_variable</structname> provides information about + session variables. + </para> + + <table> + <title><structname>pg_variable</structname> Columns</title> + <tgroup cols="1"> + <thead> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + Column Type + </para> + <para> + Description + </para></entry> + </row> + </thead> + + <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>oid</structfield> <type>oid</type> + </para> + <para> + Row identifier + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>vartype</structfield> <type>oid</type> + (references <link linkend="catalog-pg-type"><structname>pg_type</structname></link>.<structfield>oid</structfield>) + </para> + <para> + The OID of the variable's data type + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varcreate_lsn</structfield> <type>XLogRecPtr</type> + </para> + <para> + LSN of the transaction where variable was created. It is used + (in combination with <structfield>oid</structfield>) as everytime + unique identifier. Only <structfield>oid</structfield> cannot be + used for this purpose, because unused <structfield>oid</structfield> + can be reused. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varname</structfield> <type>name</type> + </para> + <para> + Name of the session variable + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varnamespace</structfield> <type>oid</type> + (references <link linkend="catalog-pg-namespace"><structname>pg_namespace</structname></link>.<structfield>oid</structfield>) + </para> + <para> + The OID of the namespace that contains this variable + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varowner</structfield> <type>oid</type> + (references <link linkend="catalog-pg-authid"><structname>pg_authid</structname></link>.<structfield>oid</structfield>) + </para> + <para> + Owner of the variable + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>vartypmod</structfield> <type>int4</type> + </para> + <para> + <structfield>vartypmod</structfield> records type-specific data + supplied at variable creation time (for example, the maximum + length of a <type>varchar</type> column). It is passed to + type-specific input functions and length coercion functions. + The value will generally be -1 for types that do not need <structfield>vartypmod</structfield>. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varisnotnull</structfield> <type>boolean</type> + </para> + <para> + True if the session variable doesn't allow null value. The default value is false. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varisimmutable</structfield> <type>boolean</type> + </para> + <para> + True if the variable is <link linkend="sql-createvariable-immutable">immutable</link> (cannot be modified). The default value is false. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>vareoxaction</structfield> <type>char</type> + </para> + <para> + Action performed at end of transaction: + <literal>n</literal> = no action, <literal>d</literal> = drop the variable, + <literal>r</literal> = reset the variable to its default value. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varcollation</structfield> <type>oid</type> + (references <link linkend="catalog-pg-collation"><structname>pg_collation</structname></link>.<structfield>oid</structfield>) + </para> + <para> + The defined collation of the variable, or zero if the variable is + not of a collatable data type. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varacl</structfield> <type>aclitem[]</type> + </para> + <para> + Access privileges; see + <xref linkend="sql-grant"/> and + <xref linkend="sql-revoke"/> + for details + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>vardefexpr</structfield> <type>pg_node_tree</type> + </para> + <para> + The internal representation of the variable default value + </para></entry> + </row> + </tbody> + </tgroup> + </table> + </sect1> </chapter> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 2073bafa1f..5a70621082 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -10443,6 +10443,66 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir' </listitem> </varlistentry> + <varlistentry id="guc-session-variables-ambiguity-warning" xreflabel="session_variables_ambiguity_warning"> + <term><varname>session_variables_ambiguity_warning</varname> (<type>boolean</type>) + <indexterm> + <primary><varname>session_variables_ambiguity_warning</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + When on, a warning is raised when any identifier in a query could be + used as both a column identifier, routine variable or a session + variable identifier. The default is <literal>off</literal>. + </para> + <para> + Session variables can be shadowed by column references in a query, this + is an expected behavior. Previously working queries shouldn't error out + by creating any session variable, so session variables are always shadowed + if an identifier is ambiguous. Variables should be referenced using + anunambiguous identifier without any possibility for a collision with + identifier of other database objects (column names or record fields names). + The warning messages emitted when enabling <varname>session_variables_ambiguity_warning</varname> + can help finding such identifier collision. +<programlisting> +CREATE TABLE foo(a int); +INSERT INTO foo VALUES(10); +CREATE VARIABLE a int; +LET a = 100; +SELECT a FROM foo; +</programlisting> + +<screen> + a +---- + 10 +(1 row) +</screen> + +<programlisting> +SET session_variables_ambiguity_warning TO on; +SELECT a FROM foo; +</programlisting> + +<screen> +WARNING: session variable "a" is shadowed +LINE 1: SELECT a FROM foo; + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. + a +---- + 10 +(1 row) +</screen> + </para> + <para> + This feature can significantly increase log size, so it's disabled by + default. For testing or development environments it's recommended to + enable it if you use session variables. + </para> + </listitem> + </varlistentry> + <varlistentry id="guc-standard-conforming-strings" xreflabel="standard_conforming_strings"> <term><varname>standard_conforming_strings</varname> (<type>boolean</type>) <indexterm><primary>strings</primary><secondary>standard conforming</secondary></indexterm> diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 458b89c0cb..e953c588f3 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -5112,6 +5112,70 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; </para> </sect1> + <sect1 id="ddl-session-variables"> + <title>Session Variables</title> + + <indexterm zone="ddl-session-variables"> + <primary>Session variables</primary> + </indexterm> + + <indexterm> + <primary>session variable</primary> + </indexterm> + + <para> + Session variables are database objects that can hold a value. + Session variables, like relations, exist within a schema and their access + is controlled via <command>GRANT</command> and <command>REVOKE</command> + commands. A session variable can be created by the <command>CREATE + VARIABLE</command> command. + </para> + + <para> + The value of a session variable is set with the <command>LET</command> SQL + command. While session variables share properties with tables, their value + cannot be updated with an <command>UPDATE</command> command. The value of a + session variable may be retrieved by the <command>SELECT</command> SQL + command. +<programlisting> +CREATE VARIABLE var1 AS date; +LET var1 = current_date; +SELECT var1; +</programlisting> + + or + +<programlisting> +CREATE VARIABLE public.current_user_id AS integer; +GRANT READ ON VARIABLE public.current_user_id TO PUBLIC; +LET current_user_id = (SELECT id FROM users WHERE usename = session_user); +SELECT current_user_id; +</programlisting> + </para> + + <para> + The value of a session variable is local to the current session. Retrieving + a variable's value returns either a <literal>NULL</literal> or a default + value, unless its value has been set to something else in the current + session using the <command>LET</command> command. The content of a variable + is not transactional. This is the same as regular variables in PL languages. + The session variables can be persistent or can be temporary. In both cases, + the content of session variables is temporary and not shared (like an + content of temporary tables). + </para> + + <para> + The session variables can be shadowed by column references in a query. When + a query contains identifiers or qualified identifiers that could be used as + both a session variable identifiers and as column identifier, then the + column identifier is preferred every time. Warnings can be emitted when + this situation happens by enabling configuration parameter <xref + linkend="guc-session-variables-ambiguity-warning"/>. User can explicitly + qualify the source object by syntax <literal>table.column</literal> or + <literal>variable.column</literal>. + </para> + </sect1> + <sect1 id="ddl-others"> <title>Other Database Objects</title> diff --git a/doc/src/sgml/event-trigger.sgml b/doc/src/sgml/event-trigger.sgml index 3b6a5361b3..cac5f9ff94 100644 --- a/doc/src/sgml/event-trigger.sgml +++ b/doc/src/sgml/event-trigger.sgml @@ -405,6 +405,14 @@ <entry align="center"><literal>-</literal></entry> <entry align="left"></entry> </row> + <row> + <entry align="left"><literal>ALTER VARIABLE</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>-</literal></entry> + <entry align="center"><literal>-</literal></entry> + <entry align="left"></entry> + </row> <row> <entry align="left"><literal>ALTER VIEW</literal></entry> <entry align="center"><literal>X</literal></entry> @@ -693,6 +701,14 @@ <entry align="center"><literal>-</literal></entry> <entry align="left"></entry> </row> + <row> + <entry align="left"><literal>CREATE VARIABLE</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>-</literal></entry> + <entry align="center"><literal>-</literal></entry> + <entry align="left"></entry> + </row> <row> <entry align="left"><literal>CREATE VIEW</literal></entry> <entry align="center"><literal>X</literal></entry> @@ -981,6 +997,14 @@ <entry align="center"><literal>-</literal></entry> <entry align="left"></entry> </row> + <row> + <entry align="left"><literal>DROP VARIABLE</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>-</literal></entry> + <entry align="left"></entry> + </row> <row> <entry align="left"><literal>DROP VIEW</literal></entry> <entry align="center"><literal>X</literal></entry> diff --git a/doc/src/sgml/glossary.sgml b/doc/src/sgml/glossary.sgml index fe8def41d0..44cfa36b96 100644 --- a/doc/src/sgml/glossary.sgml +++ b/doc/src/sgml/glossary.sgml @@ -1598,6 +1598,22 @@ </glossdef> </glossentry> + <glossentry id="glossary-session-variable"> + <glossterm>Session variable</glossterm> + <glossdef> + <para> + A persistent database object that holds a value in session memory. This + memory is not shared across sessions, and after session end, this memory + (the value) is released. The access (read or write) to session variables + is controlled by access rights similarly to other database object access + rights. + </para> + <para> + For more information, see <xref linkend="ddl-session-variables"/>. + </para> + </glossdef> + </glossentry> + <glossentry id="glossary-shared-memory"> <glossterm>Shared memory</glossterm> <glossdef> diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml index f55e901c7e..0f93647382 100644 --- a/doc/src/sgml/plpgsql.sgml +++ b/doc/src/sgml/plpgsql.sgml @@ -5970,6 +5970,18 @@ $$ LANGUAGE plpgsql STRICT IMMUTABLE; </programlisting> </para> </sect3> + + <sect3 id="plpgsql-session-variables"> + <title><command>Session variables</command></title> + + <para> + The <application>PL/pgSQL</application> language has no packages, and + therefore no package variables or package constants. + <productname>PostgreSQL</productname> has session variables and immutable + session variables. Session variables can be created by <command>CREATE + VARIABLE</command>, as described in <xref linkend="sql-createvariable"/>. + </para> + </sect3> </sect2> <sect2 id="plpgsql-porting-appendix"> diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml index 54b5f22d6e..fa295e5d77 100644 --- a/doc/src/sgml/ref/allfiles.sgml +++ b/doc/src/sgml/ref/allfiles.sgml @@ -47,6 +47,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY alterType SYSTEM "alter_type.sgml"> <!ENTITY alterUser SYSTEM "alter_user.sgml"> <!ENTITY alterUserMapping SYSTEM "alter_user_mapping.sgml"> +<!ENTITY alterVariable SYSTEM "alter_variable.sgml"> <!ENTITY alterView SYSTEM "alter_view.sgml"> <!ENTITY analyze SYSTEM "analyze.sgml"> <!ENTITY begin SYSTEM "begin.sgml"> @@ -99,6 +100,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY createType SYSTEM "create_type.sgml"> <!ENTITY createUser SYSTEM "create_user.sgml"> <!ENTITY createUserMapping SYSTEM "create_user_mapping.sgml"> +<!ENTITY createVariable SYSTEM "create_variable.sgml"> <!ENTITY createView SYSTEM "create_view.sgml"> <!ENTITY deallocate SYSTEM "deallocate.sgml"> <!ENTITY declare SYSTEM "declare.sgml"> @@ -147,6 +149,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY dropType SYSTEM "drop_type.sgml"> <!ENTITY dropUser SYSTEM "drop_user.sgml"> <!ENTITY dropUserMapping SYSTEM "drop_user_mapping.sgml"> +<!ENTITY dropVariable SYSTEM "drop_variable.sgml"> <!ENTITY dropView SYSTEM "drop_view.sgml"> <!ENTITY end SYSTEM "end.sgml"> <!ENTITY execute SYSTEM "execute.sgml"> @@ -155,6 +158,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY grant SYSTEM "grant.sgml"> <!ENTITY importForeignSchema SYSTEM "import_foreign_schema.sgml"> <!ENTITY insert SYSTEM "insert.sgml"> +<!ENTITY let SYSTEM "let.sgml"> <!ENTITY listen SYSTEM "listen.sgml"> <!ENTITY load SYSTEM "load.sgml"> <!ENTITY lock SYSTEM "lock.sgml"> diff --git a/doc/src/sgml/ref/alter_default_privileges.sgml b/doc/src/sgml/ref/alter_default_privileges.sgml index a33461fbc2..a4cc0f8907 100644 --- a/doc/src/sgml/ref/alter_default_privileges.sgml +++ b/doc/src/sgml/ref/alter_default_privileges.sgml @@ -50,6 +50,10 @@ GRANT { USAGE | CREATE | ALL [ PRIVILEGES ] } ON SCHEMAS TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] +GRANT { SELECT | UPDATE | ALL [ PRIVILEGES ] } + ON VARIABLES + TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] + REVOKE [ GRANT OPTION FOR ] { { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER | MAINTAIN } [, ...] | ALL [ PRIVILEGES ] } @@ -81,6 +85,12 @@ REVOKE [ GRANT OPTION FOR ] ON SCHEMAS FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ CASCADE | RESTRICT ] + +REVOKE [ GRANT OPTION FOR ] + { { SELECT | UPDATE } [, ...] | ALL [ PRIVILEGES ] } + ON VARIABLES + FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] + [ CASCADE | RESTRICT ] </synopsis> </refsynopsisdiv> @@ -92,14 +102,14 @@ REVOKE [ GRANT OPTION FOR ] that will be applied to objects created in the future. (It does not affect privileges assigned to already-existing objects.) Currently, only the privileges for schemas, tables (including views and foreign - tables), sequences, functions, and types (including domains) can be - altered. For this command, functions include aggregates and procedures. - The words <literal>FUNCTIONS</literal> and <literal>ROUTINES</literal> are - equivalent in this command. (<literal>ROUTINES</literal> is preferred - going forward as the standard term for functions and procedures taken - together. In earlier PostgreSQL releases, only the - word <literal>FUNCTIONS</literal> was allowed. It is not possible to set - default privileges for functions and procedures separately.) + tables), sequences, functions, types (including domains) and session + variables can be altered. For this command, functions include aggregates + and procedures. The words <literal>FUNCTIONS</literal> and + <literal>ROUTINES</literal> are equivalent in this command. + (<literal>ROUTINES</literal> is preferred going forward as the standard term + for functions and procedures taken together. In earlier PostgreSQL + releases, only the word <literal>FUNCTIONS</literal> was allowed. It is not + possible to set default privileges for functions and procedures separately.) </para> <para> diff --git a/doc/src/sgml/ref/alter_variable.sgml b/doc/src/sgml/ref/alter_variable.sgml new file mode 100644 index 0000000000..d2036351e5 --- /dev/null +++ b/doc/src/sgml/ref/alter_variable.sgml @@ -0,0 +1,179 @@ +<!-- +doc/src/sgml/ref/alter_variable.sgml +PostgreSQL documentation +--> + +<refentry id="sql-altervariable"> + <indexterm zone="sql-altervariable"> + <primary>ALTER VARIABLE</primary> + </indexterm> + + <indexterm> + <primary>session variable</primary> + <secondary>altering</secondary> + </indexterm> + + <refmeta> + <refentrytitle>ALTER VARIABLE</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo>SQL - Language Statements</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ALTER VARIABLE</refname> + <refpurpose> + change the definition of a session variable + </refpurpose> + </refnamediv> + + <refsynopsisdiv> +<synopsis> +ALTER VARIABLE <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable class="parameter">new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER } +ALTER VARIABLE <replaceable class="parameter">name</replaceable> RENAME TO <replaceable class="parameter">new_name</replaceable> +ALTER VARIABLE <replaceable class="parameter">name</replaceable> SET SCHEMA <replaceable class="parameter">new_schema</replaceable> +</synopsis> + </refsynopsisdiv> + + <refsect1> + <title>Description</title> + + <para> + The <command>ALTER VARIABLE</command> command changes the definition of an + existing session variable. There are several subforms: + + <variablelist> + <varlistentry> + <term><literal>OWNER</literal></term> + <listitem> + <para> + This form changes the owner of the session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>RENAME</literal></term> + <listitem> + <para> + This form changes the name of the session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>SET SCHEMA</literal></term> + <listitem> + <para> + This form moves the session variable into another schema. + </para> + </listitem> + </varlistentry> + + </variablelist> + </para> + + <para> + Only the owner or a superuser is allowed to alter a session variable. + In order to move a session variable from one schema to another, the user + must also have the <literal>CREATE</literal> privilege on the new schema (or + be a superuser). + + In order to move the session variable ownership from one role to another, + the user must also be a direct or indirect member of the new + owning role, and that role must have the <literal>CREATE</literal> privilege + on the session variable's schema (or be a superuser). These restrictions + enforce that altering the owner doesn't do anything you couldn't do by + dropping and recreating the session variable. + </para> + </refsect1> + + <refsect1> + <title>Parameters</title> + + <para> + <variablelist> + <varlistentry> + <term><replaceable class="parameter">name</replaceable></term> + <listitem> + <para> + The name (possibly schema-qualified) of the existing session variable + to alter. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">new_name</replaceable></term> + <listitem> + <para> + The new name for the session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">new_owner</replaceable></term> + <listitem> + <para> + The user name of the new owner of the session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">new_schema</replaceable></term> + <listitem> + <para> + The new schema for the session variable. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + </refsect1> + + <refsect1> + <title>Examples</title> + + <para> + To rename a session variable: +<programlisting> +ALTER VARIABLE foo RENAME TO boo; +</programlisting> + </para> + + <para> + To change the owner of the session variable <literal>boo</literal> to + <literal>joe</literal>: +<programlisting> +ALTER VARIABLE boo OWNER TO joe; +</programlisting> + </para> + + <para> + To change the schema of the session variable <literal>boo</literal> to + <literal>private</literal>: +<programlisting> +ALTER VARIABLE boo SET SCHEMA private; +</programlisting> + </para> + </refsect1> + + <refsect1> + <title>Compatibility</title> + + <para> + Session variables and this command in particular are a PostgreSQL extension. + </para> + </refsect1> + + <refsect1 id="sql-altervariable-see-also"> + <title>See Also</title> + + <simplelist type="inline"> + <member><xref linkend="sql-createvariable"/></member> + <member><xref linkend="sql-dropvariable"/></member> + <member><xref linkend="sql-let"/></member> + </simplelist> + </refsect1> +</refentry> diff --git a/doc/src/sgml/ref/comment.sgml b/doc/src/sgml/ref/comment.sgml index 5b43c56b13..21cd80818f 100644 --- a/doc/src/sgml/ref/comment.sgml +++ b/doc/src/sgml/ref/comment.sgml @@ -65,6 +65,7 @@ COMMENT ON TRANSFORM FOR <replaceable>type_name</replaceable> LANGUAGE <replaceable>lang_name</replaceable> | TRIGGER <replaceable class="parameter">trigger_name</replaceable> ON <replaceable class="parameter">table_name</replaceable> | TYPE <replaceable class="parameter">object_name</replaceable> | + VARIABLE <replaceable class="parameter">object_name</replaceable> | VIEW <replaceable class="parameter">object_name</replaceable> } IS { <replaceable class="parameter">string_literal</replaceable> | NULL } diff --git a/doc/src/sgml/ref/create_schema.sgml b/doc/src/sgml/ref/create_schema.sgml index ed69298ccc..a834c876bc 100644 --- a/doc/src/sgml/ref/create_schema.sgml +++ b/doc/src/sgml/ref/create_schema.sgml @@ -103,9 +103,10 @@ CREATE SCHEMA IF NOT EXISTS AUTHORIZATION <replaceable class="parameter">role_sp schema. Currently, only <command>CREATE TABLE</command>, <command>CREATE VIEW</command>, <command>CREATE INDEX</command>, <command>CREATE SEQUENCE</command>, <command>CREATE - TRIGGER</command> and <command>GRANT</command> are accepted as clauses - within <command>CREATE SCHEMA</command>. Other kinds of objects may - be created in separate commands after the schema is created. + TRIGGER</command>, <command>GRANT</command> and <command>CREATE + VARIABLE</command> are accepted as clauses within <command>CREATE + SCHEMA</command>. Other kinds of objects may be created in separate + commands after the schema is created. </para> </listitem> </varlistentry> diff --git a/doc/src/sgml/ref/create_variable.sgml b/doc/src/sgml/ref/create_variable.sgml new file mode 100644 index 0000000000..72914e133d --- /dev/null +++ b/doc/src/sgml/ref/create_variable.sgml @@ -0,0 +1,223 @@ +<!-- +doc/src/sgml/ref/create_variable.sgml +PostgreSQL documentation +--> + +<refentry id="sql-createvariable"> + <indexterm zone="sql-createvariable"> + <primary>CREATE VARIABLE</primary> + </indexterm> + + <indexterm> + <primary>session variable</primary> + <secondary>defining</secondary> + </indexterm> + + <refmeta> + <refentrytitle>CREATE VARIABLE</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo>SQL - Language Statements</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>CREATE VARIABLE</refname> + <refpurpose>define a session variable</refpurpose> + </refnamediv> + + <refsynopsisdiv> +<synopsis> +CREATE [ { TEMPORARY | TEMP } ] [ IMMUTABLE ] VARIABLE [ IF NOT EXISTS ] <replaceable class="parameter">name</replaceable> [ AS ] <replaceable class="parameter">data_type</replaceable> ] [ COLLATE <replaceable class="parameter">collation</replaceable> ] + [ NOT NULL ] [ DEFAULT <replaceable class="parameter">default_expr</replaceable> ] [ { ON COMMIT DROP | ON TRANSACTION END RESET } ] +</synopsis> + </refsynopsisdiv> + <refsect1> + <title>Description</title> + + <para> + The <command>CREATE VARIABLE</command> command creates a session variable. + Session variables, like relations, exist within a schema and their access is + controlled via <command>GRANT</command> and <command>REVOKE</command> + commands. + </para> + + <para> + The value of a session variable is local to the current session. Retrieving + a session variable's value returns either a NULL or a default value, unless + its value is set to something else in the current session with a LET + command. The content of a session variable is not transactional. This is the + same as regular variables in PL languages. + </para> + + <para> + Session variables are retrieved by the <command>SELECT</command> SQL + command. Their value is set with the <command>LET</command> SQL command. + While session variables share properties with tables, their value cannot be + changed with an <command>UPDATE</command> command. + </para> + + <note> + <para> + Inside a query or an expression, the session variable can be shadowed by + column or by routine's variable or routine argument. Such collisions of + identifiers can be resolved by using qualified identifiers. Session variables + can use schema name, columns can use table aliases, routine variables + can use block labels, and routine arguments can use the routine name. + </para> + </note> + </refsect1> + + <refsect1> + <title>Parameters</title> + + <variablelist> + <varlistentry id="sql-createvariable-immutable"> + <term><literal>IMMUTABLE</literal></term> + <listitem> + <para> + The assigned value of the session variable can not be changed. + Only if the session variable doesn't have a default value, a single + initialization is allowed using the <command>LET</command> command. Once + done, no further change is allowed until end of transaction + if the session variable was created with clause <literal>ON TRANSACTION + END RESET</literal>, or until reset of all session variables by + <command>DISCARD VARIABLES</command>, or until reset of all session + objects by command <command>DISCARD ALL</command>. + </para> + + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-if-not-exists"> + <term><literal>IF NOT EXISTS</literal></term> + <listitem> + <para> + Do not throw an error if the name already exists. A notice is issued in + this case. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-name"> + <term><replaceable class="parameter">name</replaceable></term> + <listitem> + <para> + The name, optionally schema-qualified, of the session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-data_type"> + <term><replaceable class="parameter">data_type</replaceable></term> + <listitem> + <para> + The name, optionally schema-qualified, of the data type of the session + variable. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-collate"> + <term><literal>COLLATE <replaceable>collation</replaceable></literal></term> + <listitem> + <para> + The <literal>COLLATE</literal> clause assigns a collation to the session + variable (which must be of a collatable data type). If not specified, + the data type's default collation is used. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-not-null"> + <term><literal>NOT NULL</literal></term> + <listitem> + <para> + The <literal>NOT NULL</literal> clause forbids setting the session + variable to a null value. A session variable created as NOT NULL and + without an explicitly declared default value cannot be read until it is + initialized by a LET command. This requires the user to explicitly + initialize the session variable content before reading it, otherwise an + error will be thrown. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-default"> + <term><literal>DEFAULT <replaceable>default_expr</replaceable></literal></term> + <listitem> + <para> + The <literal>DEFAULT</literal> clause can be used to assign a default + value to a session variable. This expression is evaluated when the session + variable is first accessed for reading and had not yet been assigned a value. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-on-commit-drop"> + <term><literal>ON COMMIT DROP</literal></term> + <listitem> + <para> + The <literal>ON COMMIT DROP</literal> clause specifies the behaviour of a + temporary session variable at transaction commit. With this clause, the + session variable is dropped at commit time. The clause is only allowed + for temporary variables. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-on-transaction-end-reset"> + <term><literal>ON TRANSACTION END RESET</literal></term> + <listitem> + <para> + The <literal>ON TRANSACTION END RESET</literal> clause causes the session + variable to be reset to its default value when the transaction is committed + or rolled back. + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect1> + + <refsect1> + <title>Notes</title> + + <para> + Use the <command>DROP VARIABLE</command> command to remove a session + variable. + </para> + </refsect1> + + <refsect1> + <title>Examples</title> + + <para> + Create an date session variable <literal>var1</literal>: +<programlisting> +CREATE VARIABLE var1 AS date; +LET var1 = current_date; +SELECT var1; +</programlisting> + </para> + + </refsect1> + + <refsect1> + <title>Compatibility</title> + + <para> + The <command>CREATE VARIABLE</command> command is a + <productname>PostgreSQL</productname> extension. + </para> + </refsect1> + + <refsect1> + <title>See Also</title> + + <simplelist type="inline"> + <member><xref linkend="sql-altervariable"/></member> + <member><xref linkend="sql-dropvariable"/></member> + <member><xref linkend="sql-let"/></member> + </simplelist> + </refsect1> + +</refentry> diff --git a/doc/src/sgml/ref/discard.sgml b/doc/src/sgml/ref/discard.sgml index bf44c523ca..6f90672afa 100644 --- a/doc/src/sgml/ref/discard.sgml +++ b/doc/src/sgml/ref/discard.sgml @@ -21,7 +21,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> -DISCARD { ALL | PLANS | SEQUENCES | TEMPORARY | TEMP } +DISCARD { ALL | PLANS | SEQUENCES | TEMPORARY | TEMP | VARIABLES } </synopsis> </refsynopsisdiv> @@ -75,6 +75,17 @@ DISCARD { ALL | PLANS | SEQUENCES | TEMPORARY | TEMP } </listitem> </varlistentry> + <varlistentry> + <term><literal>VARIABLES</literal></term> + <listitem> + <para> + Resets the value of all session variables. If a variable + is later reused, it is re-initialized to either + <literal>NULL</literal> or its default value. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>ALL</literal></term> <listitem> @@ -93,6 +104,7 @@ SELECT pg_advisory_unlock_all(); DISCARD PLANS; DISCARD TEMP; DISCARD SEQUENCES; +DISCARD VARIABLES; </programlisting></para> </listitem> </varlistentry> diff --git a/doc/src/sgml/ref/drop_variable.sgml b/doc/src/sgml/ref/drop_variable.sgml new file mode 100644 index 0000000000..67988b5fcd --- /dev/null +++ b/doc/src/sgml/ref/drop_variable.sgml @@ -0,0 +1,118 @@ +<!-- +doc/src/sgml/ref/drop_variable.sgml +PostgreSQL documentation +--> + +<refentry id="sql-dropvariable"> + <indexterm zone="sql-dropvariable"> + <primary>DROP VARIABLE</primary> + </indexterm> + + <indexterm> + <primary>session variable</primary> + <secondary>removing</secondary> + </indexterm> + + <refmeta> + <refentrytitle>DROP VARIABLE</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo>SQL - Language Statements</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>DROP VARIABLE</refname> + <refpurpose>remove a session variable</refpurpose> + </refnamediv> + + <refsynopsisdiv> +<synopsis> +DROP VARIABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable> [, ...] [ CASCADE | RESTRICT ] +</synopsis> + </refsynopsisdiv> + + <refsect1> + <title>Description</title> + + <para> + <command>DROP VARIABLE</command> removes a session variable. + A session variable can only be removed by its owner or a superuser. + </para> + </refsect1> + + <refsect1> + <title>Parameters</title> + + <variablelist> + <varlistentry> + <term><literal>IF EXISTS</literal></term> + <listitem> + <para> + Do not throw an error if the session variable does not exist. A notice is + issued in this case. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">name</replaceable></term> + <listitem> + <para> + The name, optionally schema-qualified, of a session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>CASCADE</literal></term> + <listitem> + <para> + Automatically drop objects that depend on the session variable (such as + views), and in turn all objects that depend on those objects + (see <xref linkend="ddl-depend"/>). + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>RESTRICT</literal></term> + <listitem> + <para> + Refuse to drop the session variable if any objects depend on it. This is + the default. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect1> + + <refsect1> + <title>Examples</title> + + <para> + To remove the session variable <literal>var1</literal>: + +<programlisting> +DROP VARIABLE var1; +</programlisting></para> + </refsect1> + + <refsect1> + <title>Compatibility</title> + + <para> + The <command>DROP VARIABLE</command> command is a + <productname>PostgreSQL</productname> extension. + </para> + </refsect1> + + <refsect1> + <title>See Also</title> + + <simplelist type="inline"> + <member><xref linkend="sql-altervariable"/></member> + <member><xref linkend="sql-createvariable"/></member> + <member><xref linkend="sql-let"/></member> + </simplelist> + </refsect1> + +</refentry> diff --git a/doc/src/sgml/ref/grant.sgml b/doc/src/sgml/ref/grant.sgml index 35bf0332c8..ba2a497780 100644 --- a/doc/src/sgml/ref/grant.sgml +++ b/doc/src/sgml/ref/grant.sgml @@ -101,6 +101,12 @@ GRANT <replaceable class="parameter">role_name</replaceable> [, ...] TO <replace [ WITH { ADMIN | INHERIT | SET } { OPTION | TRUE | FALSE } ] [ GRANTED BY <replaceable class="parameter">role_specification</replaceable> ] +GRANT { SELECT | UPDATE | ALL [ PRIVILEGES ] } + ON VARIABLE <replaceable>variable_name</replaceable> [, ...] + | ALL VARIABLES IN SCHEMA <replaceable class="parameter">schema_name</replaceable> [, ...] } + TO <replaceable class="parameter">role_specification</replaceable> [, ...] [ WITH GRANT OPTION ] + [ GRANTED BY <replaceable class="parameter">role_specification</replaceable> ] + <phrase>where <replaceable class="parameter">role_specification</replaceable> can be:</phrase> [ GROUP ] <replaceable class="parameter">role_name</replaceable> diff --git a/doc/src/sgml/ref/let.sgml b/doc/src/sgml/ref/let.sgml new file mode 100644 index 0000000000..c5d1e4d962 --- /dev/null +++ b/doc/src/sgml/ref/let.sgml @@ -0,0 +1,109 @@ +<!-- +doc/src/sgml/ref/let.sgml +PostgreSQL documentation +--> + +<refentry id="sql-let"> + <indexterm zone="sql-let"> + <primary>LET</primary> + </indexterm> + + <indexterm> + <primary>session variable</primary> + <secondary>changing</secondary> + </indexterm> + + <refmeta> + <refentrytitle>LET</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo>SQL - Language Statements</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>LET</refname> + <refpurpose>change a session variable's value</refpurpose> + </refnamediv> + + <refsynopsisdiv> +<synopsis> +LET <replaceable class="parameter">session_variable</replaceable> = <replaceable class="parameter">sql_expression</replaceable> +LET <replaceable class="parameter">session_variable</replaceable> = DEFAULT +</synopsis> + </refsynopsisdiv> + + <refsect1> + <title>Description</title> + + <para> + The <command>LET</command> command assigns a value to the specified session + variable. + </para> + + </refsect1> + + <refsect1> + <title>Parameters</title> + + <variablelist> + <varlistentry> + <term><literal>session_variable</literal></term> + <listitem> + <para> + The name of the session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>sql_expression</literal></term> + <listitem> + <para> + An SQL expression (can be subquery in parenthesis). The result must + be of castable to the same data type as the session variable (in + implicit or assignment context). + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>DEFAULT</literal></term> + <listitem> + <para> + Reset the session variable to its default value, if that is defined. + If no explicit default value has been declared, the session variable + is set to NULL. + </para> + </listitem> + </varlistentry> + </variablelist> + + <para> + Example: +<programlisting> +CREATE VARIABLE myvar AS integer; +LET myvar = 10; +LET myvar = (SELECT sum(val) FROM tab); +LET myvar = DEFAULT; +</programlisting> + </para> + </refsect1> + + <refsect1> + <title>Compatibility</title> + + <para> + The <command>LET</command> is a <productname>PostgreSQL</productname> + extension. + </para> + </refsect1> + + <refsect1> + <title>See Also</title> + + <simplelist type="inline"> + <member><xref linkend="sql-altervariable"/></member> + <member><xref linkend="sql-createvariable"/></member> + <member><xref linkend="sql-dropvariable"/></member> + </simplelist> + </refsect1> +</refentry> diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml index 47bd7dbda0..39cb595647 100644 --- a/doc/src/sgml/ref/pg_restore.sgml +++ b/doc/src/sgml/ref/pg_restore.sgml @@ -106,6 +106,17 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>-A <replaceable class="parameter">session_variable</replaceable></option></term> + <term><option>--variable=<replaceable class="parameter">session_variable</replaceable></option></term> + <listitem> + <para> + Restore a named session variable only. Multiple session variables may + be specified with multiple <option>-A</option> switches. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-c</option></term> <term><option>--clean</option></term> diff --git a/doc/src/sgml/ref/revoke.sgml b/doc/src/sgml/ref/revoke.sgml index 8df492281a..626c0231d0 100644 --- a/doc/src/sgml/ref/revoke.sgml +++ b/doc/src/sgml/ref/revoke.sgml @@ -137,6 +137,13 @@ REVOKE [ { ADMIN | INHERIT | SET } OPTION FOR ] | CURRENT_ROLE | CURRENT_USER | SESSION_USER + +REVOKE [ GRANT OPTION FOR ] + { { SELECT | UPDATE } [, ...] | ALL [ PRIVILEGES ] } + ON VARIABLE <replaceable>variable_name</replaceable> [, ...] + | ALL VARIABLES IN SCHEMA <replaceable class="parameter">schema_name</replaceable> [, ...] } + FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] + [ CASCADE | RESTRICT ] </synopsis> </refsynopsisdiv> diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml index e11b4b6130..4256a488f8 100644 --- a/doc/src/sgml/reference.sgml +++ b/doc/src/sgml/reference.sgml @@ -75,6 +75,7 @@ &alterType; &alterUser; &alterUserMapping; + &alterVariable; &alterView; &analyze; &begin; @@ -127,6 +128,7 @@ &createType; &createUser; &createUserMapping; + &createVariable; &createView; &deallocate; &declare; @@ -175,6 +177,7 @@ &dropType; &dropUser; &dropUserMapping; + &dropVariable; &dropView; &end; &execute; @@ -183,6 +186,7 @@ &grant; &importForeignSchema; &insert; + &let; &listen; &load; &lock; -- 2.40.1 [text/x-patch] v20230516-1-0004-support-of-LET-command-in-PLpgSQL.patch (11.9K, ../../CAFj8pRDF_eygHWcG8Y9d73zmus=ohVup-ngP7w-m4tbtWg14KQ@mail.gmail.com/8-v20230516-1-0004-support-of-LET-command-in-PLpgSQL.patch) download | inline diff: From 8ca573013578359c367f39896c9a3580cabe630e Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Mon, 4 Apr 2022 20:32:45 +0200 Subject: [PATCH 04/10] support of LET command in PLpgSQL The support for LET command as PLpgSQL command does assigning of expression evaluation to session variable. With extra support in PLpgSQL we can implement SQL LET command like usual PostgreSQL's command (and the assigning of result to session variable is not side effect of query evaluation). --- src/backend/executor/spi.c | 3 ++ src/backend/parser/gram.y | 8 ++++ src/backend/parser/parser.c | 3 +- src/include/parser/parser.h | 6 ++- src/pl/plpgsql/src/pl_exec.c | 55 +++++++++++++++++++++++++ src/pl/plpgsql/src/pl_funcs.c | 24 +++++++++++ src/pl/plpgsql/src/pl_gram.y | 28 ++++++++++++- src/pl/plpgsql/src/pl_reserved_kwlist.h | 1 + src/pl/plpgsql/src/plpgsql.h | 14 ++++++- src/tools/pgindent/typedefs.list | 1 + 10 files changed, 139 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/spi.c b/src/backend/executor/spi.c index 256632c985..f11df000e7 100644 --- a/src/backend/executor/spi.c +++ b/src/backend/executor/spi.c @@ -2967,6 +2967,9 @@ _SPI_error_callback(void *arg) case RAW_PARSE_PLPGSQL_ASSIGN3: errcontext("PL/pgSQL assignment \"%s\"", query); break; + case RAW_PARSE_PLPGSQL_LET: + errcontext("LET statement \"%s\"", query); + break; default: errcontext("SQL statement \"%s\"", query); break; diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index deb67f755a..eb4e156078 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -812,6 +812,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %token MODE_PLPGSQL_ASSIGN1 %token MODE_PLPGSQL_ASSIGN2 %token MODE_PLPGSQL_ASSIGN3 +%token MODE_PLPGSQL_LET /* Precedence: lowest to highest */ @@ -927,6 +928,13 @@ parse_toplevel: pg_yyget_extra(yyscanner)->parsetree = list_make1(makeRawStmt((Node *) n, 0)); } + | MODE_PLPGSQL_LET LetStmt + { + LetStmt *n = (LetStmt *) $2; + n->plpgsql_mode = true; + pg_yyget_extra(yyscanner)->parsetree = + list_make1(makeRawStmt((Node *) n, 0)); + } ; /* diff --git a/src/backend/parser/parser.c b/src/backend/parser/parser.c index e17c310cc1..d58f06342a 100644 --- a/src/backend/parser/parser.c +++ b/src/backend/parser/parser.c @@ -61,7 +61,8 @@ raw_parser(const char *str, RawParseMode mode) MODE_PLPGSQL_EXPR, /* RAW_PARSE_PLPGSQL_EXPR */ MODE_PLPGSQL_ASSIGN1, /* RAW_PARSE_PLPGSQL_ASSIGN1 */ MODE_PLPGSQL_ASSIGN2, /* RAW_PARSE_PLPGSQL_ASSIGN2 */ - MODE_PLPGSQL_ASSIGN3 /* RAW_PARSE_PLPGSQL_ASSIGN3 */ + MODE_PLPGSQL_ASSIGN3, /* RAW_PARSE_PLPGSQL_ASSIGN3 */ + MODE_PLPGSQL_LET /* RAW_PARSE_PLPGSQL_LET */ }; yyextra.have_lookahead = true; diff --git a/src/include/parser/parser.h b/src/include/parser/parser.h index 8d90064d87..0548ad8370 100644 --- a/src/include/parser/parser.h +++ b/src/include/parser/parser.h @@ -33,6 +33,9 @@ * RAW_PARSE_PLPGSQL_ASSIGNn: parse a PL/pgSQL assignment statement, * and return a one-element List containing a RawStmt node. "n" * gives the number of dotted names comprising the target ColumnRef. + * + * RAW_PARSE_PLPGSQL_LET: parse a LET statement, and return a + * one-element List containing a RawStmt node. */ typedef enum { @@ -41,7 +44,8 @@ typedef enum RAW_PARSE_PLPGSQL_EXPR, RAW_PARSE_PLPGSQL_ASSIGN1, RAW_PARSE_PLPGSQL_ASSIGN2, - RAW_PARSE_PLPGSQL_ASSIGN3 + RAW_PARSE_PLPGSQL_ASSIGN3, + RAW_PARSE_PLPGSQL_LET } RawParseMode; /* Values for the backslash_quote GUC */ diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c index 4b76f7699a..583f84e8ea 100644 --- a/src/pl/plpgsql/src/pl_exec.c +++ b/src/pl/plpgsql/src/pl_exec.c @@ -24,6 +24,7 @@ #include "catalog/pg_proc.h" #include "catalog/pg_type.h" #include "commands/defrem.h" +#include "commands/session_variable.h" #include "executor/execExpr.h" #include "executor/spi.h" #include "executor/tstoreReceiver.h" @@ -328,6 +329,8 @@ static int exec_stmt_commit(PLpgSQL_execstate *estate, PLpgSQL_stmt_commit *stmt); static int exec_stmt_rollback(PLpgSQL_execstate *estate, PLpgSQL_stmt_rollback *stmt); +static int exec_stmt_let(PLpgSQL_execstate *estate, + PLpgSQL_stmt_let *let); static void plpgsql_estate_setup(PLpgSQL_execstate *estate, PLpgSQL_function *func, @@ -2119,6 +2122,10 @@ exec_stmts(PLpgSQL_execstate *estate, List *stmts) rc = exec_stmt_rollback(estate, (PLpgSQL_stmt_rollback *) stmt); break; + case PLPGSQL_STMT_LET: + rc = exec_stmt_let(estate, (PLpgSQL_stmt_let *) stmt); + break; + default: /* point err_stmt to parent, since this one seems corrupt */ estate->err_stmt = save_estmt; @@ -4998,6 +5005,54 @@ exec_stmt_rollback(PLpgSQL_execstate *estate, PLpgSQL_stmt_rollback *stmt) return PLPGSQL_RC_OK; } +/* ---------- + * exec_stmt_let Evaluate an expression and + * put the result into a session variable. + * ---------- + */ +static int +exec_stmt_let(PLpgSQL_execstate *estate, PLpgSQL_stmt_let *stmt) +{ + bool isNull; + Oid valtype; + int32 valtypmod; + Datum value; + Oid varid; + + List *plansources; + CachedPlanSource *plansource; + + value = exec_eval_expr(estate, + stmt->expr, + &isNull, + &valtype, + &valtypmod); + + /* + * Oid of target session variable is stored in Query structure. It is + * safer to read this value directly from the plan than to hold this value + * in the plpgsql context, because it's not necessary to handle + * invalidation of the cached value. Next operations are read only without + * any allocations, so we can expect that retrieving varid from Query + * should be fast. + */ + plansources = SPI_plan_get_plan_sources(stmt->expr->plan); + if (list_length(plansources) != 1) + elog(ERROR, "unexpected length of plansources of query for LET statement"); + + plansource = (CachedPlanSource *) linitial(plansources); + if (list_length(plansource->query_list) != 1) + elog(ERROR, "unexpected length of plansource of query for LET statement"); + + varid = linitial_node(Query, plansource->query_list)->resultVariable; + if (!OidIsValid(varid)) + elog(ERROR, "oid of target session variable is not valid"); + + SetSessionVariableWithSecurityCheck(varid, value, isNull); + + return PLPGSQL_RC_OK; +} + /* ---------- * exec_assign_expr Put an expression's result into a variable. * ---------- diff --git a/src/pl/plpgsql/src/pl_funcs.c b/src/pl/plpgsql/src/pl_funcs.c index f010515fdf..73142a9147 100644 --- a/src/pl/plpgsql/src/pl_funcs.c +++ b/src/pl/plpgsql/src/pl_funcs.c @@ -288,6 +288,8 @@ plpgsql_stmt_typename(PLpgSQL_stmt *stmt) return "COMMIT"; case PLPGSQL_STMT_ROLLBACK: return "ROLLBACK"; + case PLPGSQL_STMT_LET: + return "LET"; } return "unknown"; @@ -370,6 +372,7 @@ static void free_perform(PLpgSQL_stmt_perform *stmt); static void free_call(PLpgSQL_stmt_call *stmt); static void free_commit(PLpgSQL_stmt_commit *stmt); static void free_rollback(PLpgSQL_stmt_rollback *stmt); +static void free_let(PLpgSQL_stmt_let *stmt); static void free_expr(PLpgSQL_expr *expr); @@ -459,6 +462,9 @@ free_stmt(PLpgSQL_stmt *stmt) case PLPGSQL_STMT_ROLLBACK: free_rollback((PLpgSQL_stmt_rollback *) stmt); break; + case PLPGSQL_STMT_LET: + free_let((PLpgSQL_stmt_let *) stmt); + break; default: elog(ERROR, "unrecognized cmd_type: %d", stmt->cmd_type); break; @@ -713,6 +719,12 @@ free_getdiag(PLpgSQL_stmt_getdiag *stmt) { } +static void +free_let(PLpgSQL_stmt_let *stmt) +{ + free_expr(stmt->expr); +} + static void free_expr(PLpgSQL_expr *expr) { @@ -815,6 +827,7 @@ static void dump_perform(PLpgSQL_stmt_perform *stmt); static void dump_call(PLpgSQL_stmt_call *stmt); static void dump_commit(PLpgSQL_stmt_commit *stmt); static void dump_rollback(PLpgSQL_stmt_rollback *stmt); +static void dump_let(PLpgSQL_stmt_let *stmt); static void dump_expr(PLpgSQL_expr *expr); @@ -914,6 +927,9 @@ dump_stmt(PLpgSQL_stmt *stmt) case PLPGSQL_STMT_ROLLBACK: dump_rollback((PLpgSQL_stmt_rollback *) stmt); break; + case PLPGSQL_STMT_LET: + dump_let((PLpgSQL_stmt_let *) stmt); + break; default: elog(ERROR, "unrecognized cmd_type: %d", stmt->cmd_type); break; @@ -1590,6 +1606,14 @@ dump_getdiag(PLpgSQL_stmt_getdiag *stmt) printf("\n"); } +static void +dump_let(PLpgSQL_stmt_let *stmt) +{ + dump_ind(); + dump_expr(stmt->expr); + printf("\n"); +} + static void dump_expr(PLpgSQL_expr *expr) { diff --git a/src/pl/plpgsql/src/pl_gram.y b/src/pl/plpgsql/src/pl_gram.y index 6a09bfdd67..8aa6b5d7ab 100644 --- a/src/pl/plpgsql/src/pl_gram.y +++ b/src/pl/plpgsql/src/pl_gram.y @@ -195,7 +195,7 @@ static void check_raise_parameters(PLpgSQL_stmt_raise *stmt); %type <stmt> stmt_return stmt_raise stmt_assert stmt_execsql %type <stmt> stmt_dynexecute stmt_for stmt_perform stmt_call stmt_getdiag %type <stmt> stmt_open stmt_fetch stmt_move stmt_close stmt_null -%type <stmt> stmt_commit stmt_rollback +%type <stmt> stmt_commit stmt_rollback stmt_let %type <stmt> stmt_case stmt_foreach_a %type <list> proc_exceptions @@ -302,6 +302,7 @@ static void check_raise_parameters(PLpgSQL_stmt_raise *stmt); %token <keyword> K_INTO %token <keyword> K_IS %token <keyword> K_LAST +%token <keyword> K_LET %token <keyword> K_LOG %token <keyword> K_LOOP %token <keyword> K_MERGE @@ -870,6 +871,8 @@ proc_stmt : pl_block ';' { $$ = $1; } | stmt_rollback { $$ = $1; } + | stmt_let + { $$ = $1; } ; stmt_perform : K_PERFORM @@ -986,6 +989,29 @@ stmt_assign : T_DATUM } ; +stmt_let : K_LET + { + PLpgSQL_stmt_let *new; + RawParseMode pmode; + + pmode = RAW_PARSE_PLPGSQL_LET; + + new = palloc0(sizeof(PLpgSQL_stmt_let)); + new->cmd_type = PLPGSQL_STMT_LET; + new->lineno = plpgsql_location_to_lineno(@1); + new->stmtid = ++plpgsql_curr_compile->nstatements; + + /* Push back the head name to include it in the stmt */ + plpgsql_push_back_token(K_LET); + new->expr = read_sql_construct(';', 0, 0, ";", + pmode, + false, true, true, + NULL, NULL); + + $$ = (PLpgSQL_stmt *)new; + } + ; + stmt_getdiag : K_GET getdiag_area_opt K_DIAGNOSTICS getdiag_list ';' { PLpgSQL_stmt_getdiag *new; diff --git a/src/pl/plpgsql/src/pl_reserved_kwlist.h b/src/pl/plpgsql/src/pl_reserved_kwlist.h index c116abbb7a..90eb1ca8aa 100644 --- a/src/pl/plpgsql/src/pl_reserved_kwlist.h +++ b/src/pl/plpgsql/src/pl_reserved_kwlist.h @@ -40,6 +40,7 @@ PG_KEYWORD("from", K_FROM) PG_KEYWORD("if", K_IF) PG_KEYWORD("in", K_IN) PG_KEYWORD("into", K_INTO) +PG_KEYWORD("let", K_LET) PG_KEYWORD("loop", K_LOOP) PG_KEYWORD("not", K_NOT) PG_KEYWORD("null", K_NULL) diff --git a/src/pl/plpgsql/src/plpgsql.h b/src/pl/plpgsql/src/plpgsql.h index 2b4bcd1dbe..22c9d78f30 100644 --- a/src/pl/plpgsql/src/plpgsql.h +++ b/src/pl/plpgsql/src/plpgsql.h @@ -127,7 +127,8 @@ typedef enum PLpgSQL_stmt_type PLPGSQL_STMT_PERFORM, PLPGSQL_STMT_CALL, PLPGSQL_STMT_COMMIT, - PLPGSQL_STMT_ROLLBACK + PLPGSQL_STMT_ROLLBACK, + PLPGSQL_STMT_LET } PLpgSQL_stmt_type; /* @@ -520,6 +521,17 @@ typedef struct PLpgSQL_stmt_assign PLpgSQL_expr *expr; } PLpgSQL_stmt_assign; +/* + * Let statement + */ +typedef struct PLpgSQL_stmt_let +{ + PLpgSQL_stmt_type cmd_type; + int lineno; + unsigned int stmtid; + PLpgSQL_expr *expr; +} PLpgSQL_stmt_let; + /* * PERFORM statement */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index f36952ce4c..7e2fe9f36c 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1796,6 +1796,7 @@ PLpgSQL_stmt_forq PLpgSQL_stmt_fors PLpgSQL_stmt_getdiag PLpgSQL_stmt_if +PLpgSQL_stmt_let PLpgSQL_stmt_loop PLpgSQL_stmt_open PLpgSQL_stmt_perform -- 2.40.1 [text/x-patch] v20230516-1-0003-LET-command.patch (43.7K, ../../CAFj8pRDF_eygHWcG8Y9d73zmus=ohVup-ngP7w-m4tbtWg14KQ@mail.gmail.com/9-v20230516-1-0003-LET-command.patch) download | inline diff: From 01e7c7175449c8596413d2a0f8ce44749a017d60 Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Thu, 10 Nov 2022 22:03:04 +0100 Subject: [PATCH 03/10] LET command Set session variable value to result of query or result of default expression --- src/backend/commands/explain.c | 16 ++ src/backend/commands/session_variable.c | 86 +++++++ src/backend/executor/Makefile | 1 + src/backend/executor/execMain.c | 1 + src/backend/executor/meson.build | 1 + src/backend/executor/svariableReceiver.c | 214 +++++++++++++++++ src/backend/nodes/nodeFuncs.c | 10 + src/backend/optimizer/plan/setrefs.c | 38 ++- src/backend/parser/analyze.c | 292 +++++++++++++++++++++++ src/backend/parser/gram.y | 57 ++++- src/backend/parser/parse_agg.c | 4 + src/backend/parser/parse_cte.c | 8 + src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + src/backend/parser/parse_target.c | 4 +- src/backend/rewrite/rewriteHandler.c | 34 +++ src/backend/tcop/dest.c | 7 + src/backend/tcop/utility.c | 15 ++ src/backend/utils/cache/plancache.c | 12 + src/include/commands/session_variable.h | 3 + src/include/executor/svariableReceiver.h | 25 ++ src/include/nodes/parsenodes.h | 18 ++ src/include/nodes/plannodes.h | 2 +- src/include/parser/kwlist.h | 1 + src/include/parser/parse_node.h | 1 + src/include/tcop/cmdtaglist.h | 1 + src/include/tcop/dest.h | 3 +- src/tools/pgindent/typedefs.list | 2 + 28 files changed, 843 insertions(+), 20 deletions(-) create mode 100644 src/backend/executor/svariableReceiver.c create mode 100644 src/include/executor/svariableReceiver.h diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index 5334c503e1..630116a2e2 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -501,6 +501,22 @@ ExplainOneUtility(Node *utilityStmt, IntoClause *into, ExplainState *es, else ExplainDummyGroup("Notify", NULL, es); } + else if (IsA(utilityStmt, LetStmt)) + { + LetStmt *letstmt = (LetStmt *) utilityStmt; + List *rewritten; + + if (es->format == EXPLAIN_FORMAT_TEXT) + appendStringInfoString(es->str, "SET SESSION VARIABLE\n"); + else + ExplainDummyGroup("Set Session Variable", NULL, es); + + rewritten = QueryRewrite(castNode(Query, copyObject(letstmt->query))); + Assert(list_length(rewritten) == 1); + ExplainOneQuery(linitial_node(Query, rewritten), + CURSOR_OPT_PARALLEL_OK, NULL, es, + queryString, params, queryEnv); + } else { if (es->format == EXPLAIN_FORMAT_TEXT) diff --git a/src/backend/commands/session_variable.c b/src/backend/commands/session_variable.c index b4a91ea5a3..7e8455b26a 100644 --- a/src/backend/commands/session_variable.c +++ b/src/backend/commands/session_variable.c @@ -20,16 +20,20 @@ #include "catalog/namespace.h" #include "catalog/pg_variable.h" #include "commands/session_variable.h" +#include "executor/svariableReceiver.h" #include "funcapi.h" #include "miscadmin.h" #include "optimizer/optimizer.h" +#include "rewrite/rewriteHandler.h" #include "storage/lmgr.h" #include "storage/proc.h" +#include "tcop/tcopprot.h" #include "utils/builtins.h" #include "utils/datum.h" #include "utils/inval.h" #include "utils/lsyscache.h" #include "utils/memutils.h" +#include "utils/snapmgr.h" #include "utils/syscache.h" /* @@ -1240,6 +1244,88 @@ AtEOSubXact_SessionVariable(bool isCommit, } } +/* + * Assign result of evaluated expression to session variable + */ +void +ExecuteLetStmt(ParseState *pstate, + LetStmt *stmt, + ParamListInfo params, + QueryEnvironment *queryEnv, + QueryCompletion *qc) +{ + Query *query = castNode(Query, stmt->query); + List *rewritten; + DestReceiver *dest; + AclResult aclresult; + PlannedStmt *plan; + QueryDesc *queryDesc; + Oid varid = query->resultVariable; + + Assert(OidIsValid(varid)); + + /* + * Is it allowed to write to session variable? + */ + aclresult = object_aclcheck(VariableRelationId, varid, GetUserId(), ACL_UPDATE); + if (aclresult != ACLCHECK_OK) + aclcheck_error(aclresult, OBJECT_VARIABLE, get_session_variable_name(varid)); + + /* Create dest receiver for LET */ + dest = CreateDestReceiver(DestVariable); + SetVariableDestReceiverVarid(dest, varid); + + /* run rewriter - can be used for replacement of DEFAULT node */ + query = copyObject(query); + + rewritten = QueryRewrite(query); + + Assert(list_length(rewritten) == 1); + + query = linitial_node(Query, rewritten); + Assert(query->commandType == CMD_SELECT); + + /* plan the query */ + plan = pg_plan_query(query, pstate->p_sourcetext, + CURSOR_OPT_PARALLEL_OK, params); + + /* + * Use a snapshot with an updated command ID to ensure this query sees + * results of any previously executed queries. (This could only matter if + * the planner executed an allegedly-stable function that changed the + * database contents, but let's do it anyway to be parallel to the EXPLAIN + * code path.) + */ + PushCopiedSnapshot(GetActiveSnapshot()); + UpdateActiveSnapshotCommandId(); + + /* Create a QueryDesc, redirecting output to our tuple receiver */ + queryDesc = CreateQueryDesc(plan, pstate->p_sourcetext, + GetActiveSnapshot(), InvalidSnapshot, + dest, params, queryEnv, 0); + + /* call ExecutorStart to prepare the plan for execution */ + ExecutorStart(queryDesc, 0); + + /* + * run the plan to completion. The result should be only one + * row. For an check too_many_rows we need to read two rows. + */ + ExecutorRun(queryDesc, ForwardScanDirection, 2L, true); + + /* save the rowcount if we're given a qc to fill */ + if (qc) + SetQueryCompletion(qc, CMDTAG_LET, queryDesc->estate->es_processed); + + /* and clean up */ + ExecutorFinish(queryDesc); + ExecutorEnd(queryDesc); + + FreeQueryDesc(queryDesc); + + PopActiveSnapshot(); +} + /* * pg_session_variables - designed for testing * diff --git a/src/backend/executor/Makefile b/src/backend/executor/Makefile index 11118d0ce0..71248a34f2 100644 --- a/src/backend/executor/Makefile +++ b/src/backend/executor/Makefile @@ -76,6 +76,7 @@ OBJS = \ nodeWindowAgg.o \ nodeWorktablescan.o \ spi.o \ + svariableReceiver.o \ tqueue.o \ tstoreReceiver.o diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index a407b53038..0be24d2554 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -51,6 +51,7 @@ #include "commands/session_variable.h" #include "executor/execdebug.h" #include "executor/nodeSubplan.h" +#include "executor/svariableReceiver.h" #include "foreign/fdwapi.h" #include "jit/jit.h" #include "mb/pg_wchar.h" diff --git a/src/backend/executor/meson.build b/src/backend/executor/meson.build index 65f9457c9b..b34b383b07 100644 --- a/src/backend/executor/meson.build +++ b/src/backend/executor/meson.build @@ -64,6 +64,7 @@ backend_sources += files( 'nodeWindowAgg.c', 'nodeWorktablescan.c', 'spi.c', + 'svariableReceiver.c', 'tqueue.c', 'tstoreReceiver.c', ) diff --git a/src/backend/executor/svariableReceiver.c b/src/backend/executor/svariableReceiver.c new file mode 100644 index 0000000000..7c975cbbf9 --- /dev/null +++ b/src/backend/executor/svariableReceiver.c @@ -0,0 +1,214 @@ +/*------------------------------------------------------------------------- + * + * svariableReceiver.c + * An implementation of DestReceiver that stores the result value in + * a session variable. + * + * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/backend/executor/svariableReceiver.c + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" +#include "miscadmin.h" + +#include "access/detoast.h" +#include "executor/svariableReceiver.h" +#include "commands/session_variable.h" +#include "storage/lock.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" + +typedef struct +{ + DestReceiver pub; + Oid varid; + Oid typid; + int32 typmod; + int typlen; + int slot_offset; + int rows; +} SVariableState; + + +/* + * Prepare to receive tuples from executor. + */ +static void +svariableStartupReceiver(DestReceiver *self, int operation, TupleDesc typeinfo) +{ + SVariableState *myState = (SVariableState *) self; + int natts = typeinfo->natts; + int outcols = 0; + int i; + + /* Receiver should be initialized by SetVariableDestReceiverVarid */ + Assert(OidIsValid(myState->varid)); + + for (i = 0; i < natts; i++) + { + Form_pg_attribute attr = TupleDescAttr(typeinfo, i); + Oid typid; + Oid collid; + int32 typmod; + + if (attr->attisdropped) + continue; + + if (++outcols > 1) + continue; + + get_session_variable_type_typmod_collid(myState->varid, + &typid, + &typmod, + &collid); + + /* + * double check - the type and typmod of target variable should be + * same as type and typmod of assignment expression. It should be, the + * expression is wrapped by cast to target type and typmod. + */ + if (attr->atttypid != typid || + (attr->atttypmod >= 0 && + attr->atttypmod != typmod)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("target session variable is of type %s" + " but expression is of type %s", + format_type_with_typemod(typid, typmod), + format_type_with_typemod(attr->atttypid, + attr->atttypmod)))); + + myState->typid = attr->atttypid; + myState->typmod = attr->atttypmod; + myState->typlen = attr->attlen; + myState->slot_offset = i; + } + + if (outcols != 1) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg_plural("assignment expression returned %d column", + "assignment expression returned %d columns", + outcols, + outcols))); + + + + myState->rows = 0; +} + +/* + * Receive a tuple from the executor and store it in session variable. + */ +static bool +svariableReceiveSlot(TupleTableSlot *slot, DestReceiver *self) +{ + SVariableState *myState = (SVariableState *) self; + Datum value; + bool isnull; + bool freeval = false; + + /* Make sure the tuple is fully deconstructed */ + slot_getallattrs(slot); + + value = slot->tts_values[myState->slot_offset]; + isnull = slot->tts_isnull[myState->slot_offset]; + + if (myState->typlen == -1 && !isnull && VARATT_IS_EXTERNAL(DatumGetPointer(value))) + { + value = PointerGetDatum(detoast_external_attr((struct varlena *) + DatumGetPointer(value))); + freeval = true; + } + + myState->rows += 1; + + if (myState->rows > 1) + ereport(ERROR, + (errcode(ERRCODE_TOO_MANY_ROWS), + errmsg("expression returned more than one row"))); + + SetSessionVariable(myState->varid, value, isnull); + + if (freeval) + pfree(DatumGetPointer(value)); + + return true; +} + +/* + * Clean up at end of an executor run + */ +static void +svariableShutdownReceiver(DestReceiver *self) +{ + if (((SVariableState *) self)->rows == 0) + ereport(ERROR, + (errcode(ERRCODE_NO_DATA_FOUND), + errmsg("expression returned no rows"))); +} + +/* + * Destroy receiver when done with it + */ +static void +svariableDestroyReceiver(DestReceiver *self) +{ + pfree(self); +} + +/* + * Initially create a DestReceiver object. + */ +DestReceiver * +CreateVariableDestReceiver(void) +{ + SVariableState *self = (SVariableState *) palloc0(sizeof(SVariableState)); + + self->pub.receiveSlot = svariableReceiveSlot; + self->pub.rStartup = svariableStartupReceiver; + self->pub.rShutdown = svariableShutdownReceiver; + self->pub.rDestroy = svariableDestroyReceiver; + self->pub.mydest = DestVariable; + + /* + * Private fields will be set by SetVariableDestReceiverVarid and + * svariableStartupReceiver. + */ + return (DestReceiver *) self; +} + +/* + * Set parameters for a VariableDestReceiver. + * Should be called right after creating the DestReceiver. + */ +void +SetVariableDestReceiverVarid(DestReceiver *self, Oid varid) +{ + SVariableState *myState = (SVariableState *) self; + LOCKTAG locktag PG_USED_FOR_ASSERTS_ONLY; + + Assert(myState->pub.mydest == DestVariable); + Assert(OidIsValid(varid)); + Assert(SearchSysCacheExists1(VARIABLEOID, varid)); + +#ifdef USE_ASSERT_CHECKING + + SET_LOCKTAG_OBJECT(locktag, + MyDatabaseId, + VariableRelationId, + varid, + 0); + + Assert(LockHeldByMe(&locktag, AccessShareLock)); + +#endif + + myState->varid = varid; +} diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index fe3a113c8f..56d958cfc5 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -4049,6 +4049,16 @@ raw_expression_tree_walker_impl(Node *node, return true; } break; + case T_LetStmt: + { + LetStmt *stmt = (LetStmt *) node; + + if (WALK(stmt->target)) + return true; + if (WALK(stmt->query)) + return true; + } + break; case T_PLAssignStmt: { PLAssignStmt *stmt = (PLAssignStmt *) node; diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index f18ed8ef9a..fadec889e2 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -213,7 +213,7 @@ static List *set_windowagg_runcondition_references(PlannerInfo *root, Plan *plan); static bool pull_up_has_session_variables_walker(Node *node, PlannerInfo *root); - +static void record_plan_variable_dependency(PlannerInfo *root, Oid varid); /***************************************************************************** * @@ -2074,18 +2074,7 @@ fix_expr_common(PlannerInfo *root, Node *node) Param *p = (Param *) node; if (p->paramkind == PARAM_VARIABLE) - { - PlanInvalItem *inval_item = makeNode(PlanInvalItem); - - /* paramid is still session variable id */ - inval_item->cacheId = VARIABLEOID; - inval_item->hashValue = GetSysCacheHashValue1(VARIABLEOID, - ObjectIdGetDatum(p->paramvarid)); - - /* Append this variable to global, register dependency */ - root->glob->invalItems = lappend(root->glob->invalItems, - inval_item); - } + record_plan_variable_dependency(root, p->paramvarid); } } @@ -3576,6 +3565,25 @@ record_plan_type_dependency(PlannerInfo *root, Oid typid) } } +/* + * Record dependency on a session variable. The variable can be used as a + * session variable in expression list, or target of LET statement. + */ +static void +record_plan_variable_dependency(PlannerInfo *root, Oid varid) +{ + PlanInvalItem *inval_item = makeNode(PlanInvalItem); + + /* paramid is still session variable id */ + inval_item->cacheId = VARIABLEOID; + inval_item->hashValue = GetSysCacheHashValue1(VARIABLEOID, + ObjectIdGetDatum(varid)); + + /* Append this variable to global, register dependency */ + root->glob->invalItems = lappend(root->glob->invalItems, + inval_item); +} + /* * extract_query_dependencies * Given a rewritten, but not yet planned, query or queries @@ -3649,6 +3657,10 @@ extract_query_dependencies_walker(Node *node, PlannerInfo *context) query = UtilityContainsQuery(query->utilityStmt); if (query == NULL) return false; + + /* Record the session variable used as target of LET statement. */ + if (OidIsValid(query->resultVariable)) + record_plan_variable_dependency(context, query->resultVariable); } /* Remember if any Query has RLS quals applied by rewriter */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index c2f4f4836f..fd17b4f7c0 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -25,9 +25,12 @@ #include "postgres.h" #include "access/sysattr.h" +#include "catalog/namespace.h" #include "catalog/pg_proc.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "commands/defrem.h" +#include "commands/session_variable.h" #include "miscadmin.h" #include "nodes/makefuncs.h" #include "nodes/nodeFuncs.h" @@ -52,6 +55,7 @@ #include "utils/backend_status.h" #include "utils/builtins.h" #include "utils/guc.h" +#include "utils/lsyscache.h" #include "utils/rel.h" #include "utils/syscache.h" @@ -85,6 +89,8 @@ static Query *transformCreateTableAsStmt(ParseState *pstate, CreateTableAsStmt *stmt); static Query *transformCallStmt(ParseState *pstate, CallStmt *stmt); +static Query *transformLetStmt(ParseState *pstate, + LetStmt *stmt); static void transformLockingClause(ParseState *pstate, Query *qry, LockingClause *lc, bool pushedDown); #ifdef RAW_EXPRESSION_COVERAGE_TEST @@ -328,6 +334,7 @@ transformStmt(ParseState *pstate, Node *parseTree) case T_UpdateStmt: case T_DeleteStmt: case T_MergeStmt: + case T_LetStmt: (void) test_raw_expression_coverage(parseTree, NULL); break; default: @@ -401,6 +408,11 @@ transformStmt(ParseState *pstate, Node *parseTree) (CallStmt *) parseTree); break; + case T_LetStmt: + result = transformLetStmt(pstate, + (LetStmt *) parseTree); + break; + default: /* @@ -443,6 +455,7 @@ analyze_requires_snapshot(RawStmt *parseTree) case T_MergeStmt: case T_SelectStmt: case T_PLAssignStmt: + case T_LetStmt: result = true; break; @@ -1642,6 +1655,285 @@ transformValuesClause(ParseState *pstate, SelectStmt *stmt) return qry; } +/* + * transformLetStmt - + * transform an Let Statement + */ +static Query * +transformLetStmt(ParseState *pstate, LetStmt *stmt) +{ + Query *query; + Query *result; + List *exprList = NIL; + List *exprListCoer = NIL; + ListCell *lc; + ListCell *indirection_head = NULL; + Query *selectQuery; + Oid varid; + char *attrname = NULL; + bool not_unique; + bool is_rowtype; + Oid typid; + int32 typmod; + Oid collid; + AclResult aclresult; + List *names = NULL; + int indirection_start; + int i = 0; + + /* There can't be any outer WITH to worry about */ + Assert(pstate->p_ctenamespace == NIL); + + names = NamesFromList(stmt->target); + + /* + * The AccessShareLock is created on related session variable. The lock + * will be kept for the whole transaction. + */ + varid = IdentifyVariable(names, &attrname, ¬_unique, false); + if (not_unique) + ereport(ERROR, + (errcode(ERRCODE_AMBIGUOUS_PARAMETER), + errmsg("target \"%s\" of LET command is ambiguous", + NameListToString(names)), + parser_errposition(pstate, stmt->location))); + + if (!OidIsValid(varid)) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("session variable \"%s\" doesn't exist", + NameListToString(names)), + parser_errposition(pstate, stmt->location))); + + /* + * Calculate start of possible position of an indirection in list, + * and when it is inside the list, store pointer on first node + * of indirection. + */ + indirection_start = list_length(names) - (attrname ? 1 : 0); + if (list_length(stmt->target) > indirection_start) + indirection_head = list_nth_cell(stmt->target, indirection_start); + + get_session_variable_type_typmod_collid(varid, &typid, &typmod, &collid); + + is_rowtype = type_is_rowtype(typid); + + if (attrname && !is_rowtype) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("type \"%s\" of target session variable \"%s.%s\" is not a composite type", + format_type_be(typid), + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)), + parser_errposition(pstate, stmt->location))); + + if (stmt->set_default && attrname != NULL) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("only session variable (without attribute specification) can be set to default"), + parser_errposition(pstate, stmt->location))); + + aclresult = object_aclcheck(VariableRelationId, varid, GetUserId(), ACL_UPDATE); + if (aclresult != ACLCHECK_OK) + aclcheck_error(aclresult, OBJECT_VARIABLE, NameListToString(names)); + + pstate->p_expr_kind = EXPR_KIND_LET_TARGET; + + /* + * The LET statements suppports two syntaxes: LET var = expr and LET var = + * DEFAULT. In first case, the expression is of SelectStmt node and it is + * transformated to query (SELECT) by usual way. Second syntax should be + * transformed differently. It is more cleaner do this transformation here + * (like special case), because we don't need to enhance SelectStmt about + * fields necessary for this transformation somwehere in SelectStmt + * transformation. Isn't to necessary to uglify the SelectStmt + * transformation. This is special case of LET statement, not SelectStmt. + */ + if (stmt->set_default) + { + selectQuery = makeNode(Query); + selectQuery->commandType = CMD_SELECT; + + /* + * ResTarget(SetToDefault) -> TargetEntry(expr(SetToDefault)) The + * SetToDefault is replaced by defexpr by rewriter in RewriteQuery + * function. + */ + selectQuery->targetList = transformTargetList(pstate, + ((SelectStmt *) stmt->query)->targetList, + EXPR_KIND_LET_TARGET); + } + else + selectQuery = transformStmt(pstate, stmt->query); + + /* The grammar should have produced a SELECT */ + Assert(IsA(selectQuery, Query) && selectQuery->commandType == CMD_SELECT); + + /*---------- + * Generate an expression list for the LET that selects all the + * non-resjunk columns from the subquery. + *---------- + */ + exprList = NIL; + foreach(lc, selectQuery->targetList) + { + TargetEntry *tle = (TargetEntry *) lfirst(lc); + + if (tle->resjunk) + continue; + + exprList = lappend(exprList, tle->expr); + } + + /* don't allow multicolumn result */ + if (list_length(exprList) != 1) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg_plural("assignment expression returned %d column", + "assignment expression returned %d columns", + list_length(exprList), + list_length(exprList)), + parser_errposition(pstate, + exprLocation((Node *) exprList)))); + + exprListCoer = NIL; + + foreach(lc, exprList) + { + Expr *expr = (Expr *) lfirst(lc); + Expr *coerced_expr; + Param *param; + Oid exprtypid; + + if (IsA(expr, SetToDefault)) + { + SetToDefault *def = (SetToDefault *) expr; + + def->typeId = typid; + def->typeMod = typmod; + def->collation = collid; + } + else if (IsA(expr, Const) && ((Const *) expr)->constisnull) + { + /* use known type for NULL value */ + expr = (Expr *) makeNullConst(typid, typmod, collid); + } + + /* now we can read type of expression */ + exprtypid = exprType((Node *) expr); + + param = makeNode(Param); + param->paramkind = PARAM_VARIABLE; + param->paramvarid = varid; + param->paramtype = typid; + param->paramtypmod = typmod; + + if (indirection_head) + { + bool targetIsArray; + char *targetName; + + targetName = get_session_variable_name(varid); + targetIsArray = OidIsValid(get_element_type(typid)); + + pstate->p_hasSessionVariables = true; + + coerced_expr = (Expr *) + transformAssignmentIndirection(pstate, + (Node *) param, + targetName, + targetIsArray, + typid, + typmod, + InvalidOid, + stmt->target, + indirection_head, + (Node *) expr, + COERCION_PLPGSQL, + stmt->location); + } + else + coerced_expr = (Expr *) + coerce_to_target_type(pstate, + (Node *) expr, + exprtypid, + typid, typmod, + COERCION_ASSIGNMENT, + COERCE_IMPLICIT_CAST, + stmt->location); + + if (coerced_expr == NULL) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("variable \"%s.%s\" is of type %s," + " but expression is of type %s", + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid), + format_type_be(typid), + format_type_be(exprtypid)), + errhint("You will need to rewrite or cast the expression."), + parser_errposition(pstate, exprLocation((Node *) expr)))); + + exprListCoer = lappend(exprListCoer, coerced_expr); + } + + /* + * Generate query's target list using the computed list of expressions. + */ + query = makeNode(Query); + query->commandType = CMD_SELECT; + + foreach(lc, exprListCoer) + { + Expr *expr = (Expr *) lfirst(lc); + TargetEntry *tle; + + tle = makeTargetEntry(expr, + i + 1, + FigureColname((Node *) expr), + false); + query->targetList = lappend(query->targetList, tle); + } + + /* done building the range table and jointree */ + query->rtable = pstate->p_rtable; + query->jointree = makeFromExpr(pstate->p_joinlist, NULL); + + query->hasTargetSRFs = pstate->p_hasTargetSRFs; + query->hasSubLinks = pstate->p_hasSubLinks; + query->hasSessionVariables = pstate->p_hasSessionVariables; + + /* This is top query */ + query->canSetTag = true; + + /* + * Save target session variable id. This value is used multiple times: by + * query rewriter (for getting related defexpr), by planner (for acquiring + * AccessShareLock on target variable), and by executor (we need to know + * target variable id). + */ + query->resultVariable = varid; + + assign_query_collations(pstate, query); + + stmt->query = (Node *) query; + + /* + * When statement is executed as a PlpgSQL LET statement, then we should + * return the query because we don't want to use a utilityStmt wrapper + * node. + */ + if (stmt->plpgsql_mode) + return query; + + /* represent the command as a utility Query */ + result = makeNode(Query); + result->commandType = CMD_UTILITY; + result->utilityStmt = (Node *) stmt; + + return result; +} + /* * transformSetOperationStmt - * transforms a set-operations tree diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index c9a93a3c7a..deb67f755a 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -304,7 +304,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); DropTransformStmt DropUserMappingStmt ExplainStmt FetchStmt GrantStmt GrantRoleStmt ImportForeignSchemaStmt IndexStmt InsertStmt - ListenStmt LoadStmt LockStmt MergeStmt NotifyStmt ExplainableStmt PreparableStmt + LetStmt ListenStmt LoadStmt LockStmt MergeStmt NotifyStmt ExplainableStmt PreparableStmt CreateFunctionStmt AlterFunctionStmt ReindexStmt RemoveAggrStmt RemoveFuncStmt RemoveOperStmt RenameStmt ReturnStmt RevokeStmt RevokeRoleStmt RuleActionStmt RuleActionStmtOrEmpty RuleStmt @@ -452,6 +452,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); TriggerTransitions TriggerReferencing vacuum_relation_list opt_vacuum_relation_list drop_option_list pub_obj_list + let_target %type <node> opt_routine_body %type <groupclause> group_clause @@ -732,7 +733,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); KEY KEYS LABEL LANGUAGE LARGE_P LAST_P LATERAL_P - LEADING LEAKPROOF LEAST LEFT LEVEL LIKE LIMIT LISTEN LOAD LOCAL + LEADING LEAKPROOF LEAST LEFT LET LEVEL LIKE LIMIT LISTEN LOAD LOCAL LOCALTIME LOCALTIMESTAMP LOCATION LOCK_P LOCKED LOGGED MAPPING MATCH MATCHED MATERIALIZED MAXVALUE MERGE METHOD @@ -1067,6 +1068,7 @@ stmt: | ImportForeignSchemaStmt | IndexStmt | InsertStmt + | LetStmt | ListenStmt | RefreshMatViewStmt | LoadStmt @@ -11964,7 +11966,8 @@ ExplainableStmt: | CreateAsStmt | CreateMatViewStmt | RefreshMatViewStmt - | ExecuteStmt /* by default all are $$=$1 */ + | ExecuteStmt + | LetStmt /* by default all are $$=$1 */ ; /***************************************************************************** @@ -11994,7 +11997,8 @@ PreparableStmt: | InsertStmt | UpdateStmt | DeleteStmt - | MergeStmt /* by default all are $$=$1 */ + | MergeStmt + | LetStmt /* by default all are $$=$1 */ ; /***************************************************************************** @@ -12582,6 +12586,49 @@ opt_hold: /* EMPTY */ { $$ = 0; } | WITHOUT HOLD { $$ = 0; } ; +/***************************************************************************** + * + * QUERY: + * LET STATEMENTS + * + *****************************************************************************/ +LetStmt: LET let_target '=' a_expr + { + LetStmt *n = makeNode(LetStmt); + SelectStmt *select; + ResTarget *res; + + n->target = $2; + + select = makeNode(SelectStmt); + res = makeNode(ResTarget); + + /* Create target list for implicit query */ + res->name = NULL; + res->indirection = NIL; + res->val = (Node *) $4; + res->location = @4; + + select->targetList = list_make1(res); + n->query = (Node *) select; + + n->set_default = IsA($4, SetToDefault); + + n->location = @2; + $$ = (Node *) n; + } + ; + +let_target: + ColId opt_indirection + { + $$ = list_make1(makeString($1)); + if ($2) + $$ = list_concat($$, + check_indirection($2, yyscanner)); + } + ; + /***************************************************************************** * * QUERY: @@ -17271,6 +17318,7 @@ unreserved_keyword: | LARGE_P | LAST_P | LEAKPROOF + | LET | LEVEL | LISTEN | LOAD @@ -17855,6 +17903,7 @@ bare_label_keyword: | LEAKPROOF | LEAST | LEFT + | LET | LEVEL | LIKE | LISTEN diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 6fc03166c6..53ff67c617 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -349,6 +349,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) Assert(false); /* can't happen */ break; case EXPR_KIND_OTHER: + case EXPR_KIND_LET_TARGET: /* * Accept aggregate/grouping here; caller must throw error if @@ -955,6 +956,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_LET_TARGET: + err = _("window functions are not allowed in LET statement"); + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_cte.c b/src/backend/parser/parse_cte.c index c5b1a49725..35aa070571 100644 --- a/src/backend/parser/parse_cte.c +++ b/src/backend/parser/parse_cte.c @@ -133,6 +133,14 @@ transformWithClause(ParseState *pstate, WithClause *withClause) errmsg("MERGE not supported in WITH query"), parser_errposition(pstate, cte->location)); + /* LET is allowed by parser, but not supported. Reject for now */ + if (IsA(cte->ctequery, LetStmt)) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("LET not supported in WITH query"), + parser_errposition(pstate, cte->location)); + + for_each_cell(rest, withClause->ctes, lnext(withClause->ctes, lc)) { CommonTableExpr *cte2 = (CommonTableExpr *) lfirst(rest); diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index f40f62425e..7b7c2a16e5 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -623,6 +623,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: case EXPR_KIND_VARIABLE_DEFAULT: + case EXPR_KIND_LET_TARGET: /* okay */ break; @@ -2001,6 +2002,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_LET_TARGET: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3329,6 +3331,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_LET_TARGET: + return "LET"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index a0fdaa8605..ed5e1db03d 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2657,6 +2657,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_LET_TARGET: + err = _("set-returning functions are not allowed in LET target expression"); + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_target.c b/src/backend/parser/parse_target.c index e77b542fd7..ef5601ef4c 100644 --- a/src/backend/parser/parse_target.c +++ b/src/backend/parser/parse_target.c @@ -88,7 +88,9 @@ transformTargetEntry(ParseState *pstate, * through unmodified. (transformExpr will throw the appropriate * error if we're disallowing it.) */ - if (exprKind == EXPR_KIND_UPDATE_SOURCE && IsA(node, SetToDefault)) + if ((exprKind == EXPR_KIND_UPDATE_SOURCE || + exprKind == EXPR_KIND_LET_TARGET) + && IsA(node, SetToDefault)) expr = node; else expr = transformExpr(pstate, node, exprKind); diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index 980dc1816f..fd8ec0dbc6 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -25,6 +25,7 @@ #include "access/table.h" #include "catalog/dependency.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "commands/trigger.h" #include "executor/executor.h" #include "foreign/fdwapi.h" @@ -3701,6 +3702,39 @@ RewriteQuery(Query *parsetree, List *rewrite_events, int orig_rt_length) } } + /* + * Rewrite SetToDefault by default expression of Let statement. + */ + if (event == CMD_SELECT && OidIsValid(parsetree->resultVariable)) + { + Oid resultVariable = parsetree->resultVariable; + + if (list_length(parsetree->targetList) == 1 && + IsA(((TargetEntry *) linitial(parsetree->targetList))->expr, SetToDefault)) + { + Variable var; + TargetEntry *tle; + TargetEntry *newtle; + Expr *defexpr; + + /* Read session variable metadata with defexpr */ + InitVariable(&var, resultVariable, false); + + if (var.has_defexpr) + defexpr = (Expr *) var.defexpr; + else + defexpr = (Expr *) makeNullConst(var.typid, var.typmod, var.collation); + + tle = (TargetEntry *) linitial(parsetree->targetList); + newtle = makeTargetEntry(defexpr, + tle->resno, + pstrdup(tle->resname), + false); + + parsetree->targetList = list_make1(newtle); + } + } + /* * If the statement is an insert, update, delete, or merge, adjust its * targetlist as needed, and then fire INSERT/UPDATE/DELETE rules on it. diff --git a/src/backend/tcop/dest.c b/src/backend/tcop/dest.c index c0406e2ee5..86dbf370ac 100644 --- a/src/backend/tcop/dest.c +++ b/src/backend/tcop/dest.c @@ -37,6 +37,7 @@ #include "executor/functions.h" #include "executor/tqueue.h" #include "executor/tstoreReceiver.h" +#include "executor/svariableReceiver.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "utils/portal.h" @@ -152,6 +153,9 @@ CreateDestReceiver(CommandDest dest) case DestTupleQueue: return CreateTupleQueueDestReceiver(NULL); + + case DestVariable: + return CreateVariableDestReceiver(); } /* should never get here */ @@ -187,6 +191,7 @@ EndCommand(const QueryCompletion *qc, CommandDest dest, bool force_undecorated_o case DestSQLFunction: case DestTransientRel: case DestTupleQueue: + case DestVariable: break; } } @@ -232,6 +237,7 @@ NullCommand(CommandDest dest) case DestSQLFunction: case DestTransientRel: case DestTupleQueue: + case DestVariable: break; } } @@ -275,6 +281,7 @@ ReadyForQuery(CommandDest dest) case DestSQLFunction: case DestTransientRel: case DestTupleQueue: + case DestVariable: break; } } diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index ddc26794b2..40cc8231fb 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -242,6 +242,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree) case T_CallStmt: case T_DoStmt: + case T_LetStmt: { /* * Commands inside the DO block or the called procedure might @@ -1074,6 +1075,11 @@ standard_ProcessUtility(PlannedStmt *pstmt, break; } + case T_LetStmt: + ExecuteLetStmt(pstate, (LetStmt *) parsetree, params, + queryEnv, qc); + break; + default: /* All other statement types have event trigger support */ ProcessUtilitySlow(pstate, pstmt, queryString, @@ -2207,6 +2213,10 @@ UtilityContainsQuery(Node *parsetree) return UtilityContainsQuery(qry->utilityStmt); return qry; + case T_LetStmt: + qry = castNode(Query, ((LetStmt *) parsetree)->query); + return qry; + default: return NULL; } @@ -2405,6 +2415,10 @@ CreateCommandTag(Node *parsetree) tag = CMDTAG_SELECT; break; + case T_LetStmt: + tag = CMDTAG_LET; + break; + /* utility statements --- same whether raw or cooked */ case T_TransactionStmt: { @@ -3290,6 +3304,7 @@ GetCommandLogLevel(Node *parsetree) break; case T_PLAssignStmt: + case T_LetStmt: lev = LOGSTMT_ALL; break; diff --git a/src/backend/utils/cache/plancache.c b/src/backend/utils/cache/plancache.c index a3bea4e951..d03fe5ec4f 100644 --- a/src/backend/utils/cache/plancache.c +++ b/src/backend/utils/cache/plancache.c @@ -135,6 +135,7 @@ InitPlanCache(void) CacheRegisterSyscacheCallback(AMOPOPID, PlanCacheSysCallback, (Datum) 0); CacheRegisterSyscacheCallback(FOREIGNSERVEROID, PlanCacheSysCallback, (Datum) 0); CacheRegisterSyscacheCallback(FOREIGNDATAWRAPPEROID, PlanCacheSysCallback, (Datum) 0); + CacheRegisterSyscacheCallback(VARIABLEOID, PlanCacheObjectCallback, (Datum) 0); } /* @@ -1886,6 +1887,17 @@ ScanQueryForLocks(Query *parsetree, bool acquire) (void *) &acquire, QTW_IGNORE_RC_SUBQUERIES); } + + /* Process session variables */ + if (OidIsValid(parsetree->resultVariable)) + { + if (acquire) + LockDatabaseObject(VariableRelationId, parsetree->resultVariable, + 0, AccessShareLock); + else + UnlockDatabaseObject(VariableRelationId, parsetree->resultVariable, + 0, AccessShareLock); + } } /* diff --git a/src/include/commands/session_variable.h b/src/include/commands/session_variable.h index f442c5d3b8..ceb0d357da 100644 --- a/src/include/commands/session_variable.h +++ b/src/include/commands/session_variable.h @@ -39,4 +39,7 @@ extern void AtEOSubXact_SessionVariable(bool isCommit, SubTransactionId mySubid, SubTransactionId parentSubid); +extern void ExecuteLetStmt(ParseState *pstate, LetStmt *stmt, ParamListInfo params, + QueryEnvironment *queryEnv, QueryCompletion *qc); + #endif diff --git a/src/include/executor/svariableReceiver.h b/src/include/executor/svariableReceiver.h new file mode 100644 index 0000000000..63f6ee9b7f --- /dev/null +++ b/src/include/executor/svariableReceiver.h @@ -0,0 +1,25 @@ +/*------------------------------------------------------------------------- + * + * svariableReceiver.h + * prototypes for svariableReceiver.c + * + * + * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/executor/svariableReceiver.h + * + *------------------------------------------------------------------------- + */ + +#ifndef SVARIABLE_RECEIVER_H +#define SVARIABLE_RECEIVER_H + +#include "tcop/dest.h" + + +extern DestReceiver *CreateVariableDestReceiver(void); + +extern void SetVariableDestReceiverVarid(DestReceiver *self, Oid varid); + +#endif /* SVARIABLE_RECEIVER_H */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 1f067fc84f..b96b56fd6a 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -149,6 +149,9 @@ typedef struct Query */ int resultRelation pg_node_attr(query_jumble_ignore); + /* target variable of LET statement */ + Oid resultVariable; + /* has aggregates in tlist or havingQual */ bool hasAggs pg_node_attr(query_jumble_ignore); /* has window functions in tlist */ @@ -1918,6 +1921,21 @@ typedef struct MergeStmt WithClause *withClause; /* WITH clause */ } MergeStmt; +/* ---------------------- + * Let Statement + * ---------------------- + */ +typedef struct LetStmt +{ + NodeTag type; + List *target; /* target variable */ + Node *query; /* source expression */ + bool set_default; /* true, when set to DEFAULt is wanted */ + bool plpgsql_mode; /* true, when command will be executed + * (parsed) by plpgsql runtime */ + int location; +} LetStmt; + /* ---------------------- * Select Statement * diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index d287a80dce..69084136e7 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -50,7 +50,7 @@ typedef struct PlannedStmt NodeTag type; - CmdType commandType; /* select|insert|update|delete|merge|utility */ + CmdType commandType; /* select|let|insert|update|delete|merge|utility */ uint64 queryId; /* query identifier (copied from Query) */ diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 9f79b90251..7f2445c049 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -246,6 +246,7 @@ PG_KEYWORD("leading", LEADING, RESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("leakproof", LEAKPROOF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("least", LEAST, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("left", LEFT, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) +PG_KEYWORD("let", LET, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("level", LEVEL, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("like", LIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("limit", LIMIT, RESERVED_KEYWORD, AS_LABEL) diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h index 4e9692d05a..c3f1ccb4c0 100644 --- a/src/include/parser/parse_node.h +++ b/src/include/parser/parse_node.h @@ -82,6 +82,7 @@ typedef enum ParseExprKind EXPR_KIND_GENERATED_COLUMN, /* generation expression for a column */ EXPR_KIND_CYCLE_MARK, /* cycle mark value */ EXPR_KIND_VARIABLE_DEFAULT, /* default value for session variable */ + EXPR_KIND_LET_TARGET, /* LET assignment (should be same like UPDATE) */ } ParseExprKind; diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h index 259bdc994e..648a4af305 100644 --- a/src/include/tcop/cmdtaglist.h +++ b/src/include/tcop/cmdtaglist.h @@ -186,6 +186,7 @@ PG_CMDTAG(CMDTAG_GRANT, "GRANT", true, false, false) PG_CMDTAG(CMDTAG_GRANT_ROLE, "GRANT ROLE", false, false, false) PG_CMDTAG(CMDTAG_IMPORT_FOREIGN_SCHEMA, "IMPORT FOREIGN SCHEMA", true, false, false) PG_CMDTAG(CMDTAG_INSERT, "INSERT", false, false, true) +PG_CMDTAG(CMDTAG_LET, "LET", false, false, false) PG_CMDTAG(CMDTAG_LISTEN, "LISTEN", false, false, false) PG_CMDTAG(CMDTAG_LOAD, "LOAD", false, false, false) PG_CMDTAG(CMDTAG_LOCK_TABLE, "LOCK TABLE", false, false, false) diff --git a/src/include/tcop/dest.h b/src/include/tcop/dest.h index a7d86e7abd..e7dd749949 100644 --- a/src/include/tcop/dest.h +++ b/src/include/tcop/dest.h @@ -95,7 +95,8 @@ typedef enum DestCopyOut, /* results sent to COPY TO code */ DestSQLFunction, /* results sent to SQL-language func mgr */ DestTransientRel, /* results sent to transient relation */ - DestTupleQueue /* results sent to tuple queue */ + DestTupleQueue, /* results sent to tuple queue */ + DestVariable /* results sents to session variable */ } CommandDest; /* ---------------- diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index ad3eec1f74..f36952ce4c 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1407,6 +1407,7 @@ LargeObjectDesc LastAttnumInfo Latch LerpFunc +LetStmt LexDescr LexemeEntry LexemeHashKey @@ -2666,6 +2667,7 @@ SupportRequestRows SupportRequestSelectivity SupportRequestSimplify SupportRequestWFuncMonotonic +SVariableState SVariable SVariableData SVariableXActAction -- 2.40.1 [text/x-patch] v20230516-1-0005-DISCARD-VARIABLES-command.patch (3.2K, ../../CAFj8pRDF_eygHWcG8Y9d73zmus=ohVup-ngP7w-m4tbtWg14KQ@mail.gmail.com/10-v20230516-1-0005-DISCARD-VARIABLES-command.patch) download | inline diff: From cf543cb6ebe1d5a4369205008c2d7927393fbf5e Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Tue, 5 Apr 2022 00:13:58 +0200 Subject: [PATCH 05/10] DISCARD VARIABLES command --- src/backend/commands/discard.c | 6 ++++++ src/backend/parser/gram.y | 7 ++++++- src/backend/tcop/utility.c | 3 +++ src/include/nodes/parsenodes.h | 3 ++- src/include/tcop/cmdtaglist.h | 1 + 5 files changed, 18 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/discard.c b/src/backend/commands/discard.c index 296dc82d2e..a8189dd12c 100644 --- a/src/backend/commands/discard.c +++ b/src/backend/commands/discard.c @@ -19,6 +19,7 @@ #include "commands/discard.h" #include "commands/prepare.h" #include "commands/sequence.h" +#include "commands/session_variable.h" #include "utils/guc.h" #include "utils/portal.h" @@ -48,6 +49,10 @@ DiscardCommand(DiscardStmt *stmt, bool isTopLevel) ResetTempTableNamespace(); break; + case DISCARD_VARIABLES: + ResetSessionVariables(); + break; + default: elog(ERROR, "unrecognized DISCARD target: %d", stmt->target); } @@ -75,4 +80,5 @@ DiscardAll(bool isTopLevel) ResetPlanCache(); ResetTempTableNamespace(); ResetSequenceCaches(); + ResetSessionVariables(); } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index eb4e156078..4392491807 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -2069,7 +2069,12 @@ DiscardStmt: n->target = DISCARD_SEQUENCES; $$ = (Node *) n; } - + | DISCARD VARIABLES + { + DiscardStmt *n = makeNode(DiscardStmt); + n->target = DISCARD_VARIABLES; + $$ = (Node *) n; + } ; diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 40cc8231fb..df322e4fcb 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -2962,6 +2962,9 @@ CreateCommandTag(Node *parsetree) case DISCARD_SEQUENCES: tag = CMDTAG_DISCARD_SEQUENCES; break; + case DISCARD_VARIABLES: + tag = CMDTAG_DISCARD_VARIABLES; + break; default: tag = CMDTAG_UNKNOWN; } diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index b96b56fd6a..8948523140 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -3794,7 +3794,8 @@ typedef enum DiscardMode DISCARD_ALL, DISCARD_PLANS, DISCARD_SEQUENCES, - DISCARD_TEMP + DISCARD_TEMP, + DISCARD_VARIABLES } DiscardMode; typedef struct DiscardStmt diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h index 648a4af305..8ce00b4ea1 100644 --- a/src/include/tcop/cmdtaglist.h +++ b/src/include/tcop/cmdtaglist.h @@ -135,6 +135,7 @@ PG_CMDTAG(CMDTAG_DISCARD_ALL, "DISCARD ALL", false, false, false) PG_CMDTAG(CMDTAG_DISCARD_PLANS, "DISCARD PLANS", false, false, false) PG_CMDTAG(CMDTAG_DISCARD_SEQUENCES, "DISCARD SEQUENCES", false, false, false) PG_CMDTAG(CMDTAG_DISCARD_TEMP, "DISCARD TEMP", false, false, false) +PG_CMDTAG(CMDTAG_DISCARD_VARIABLES, "DISCARD VARIABLES", false, false, false) PG_CMDTAG(CMDTAG_DO, "DO", false, false, false) PG_CMDTAG(CMDTAG_DROP_ACCESS_METHOD, "DROP ACCESS METHOD", true, false, false) PG_CMDTAG(CMDTAG_DROP_AGGREGATE, "DROP AGGREGATE", true, false, false) -- 2.40.1 [text/x-patch] v20230516-1-0002-session-variables.patch (111.9K, ../../CAFj8pRDF_eygHWcG8Y9d73zmus=ohVup-ngP7w-m4tbtWg14KQ@mail.gmail.com/11-v20230516-1-0002-session-variables.patch) download | inline diff: From 928c06944dcbdd810180dc6faa55a565c62055da Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Sun, 13 Nov 2022 17:45:23 +0100 Subject: [PATCH 02/10] session variables Implementation storage and access routines. Session variables are stored in session memory inside dedicated hash table. Two levels of an access are implemented: API level and SQL level. Both levels are implemented in this commit. The most difficult part is cleaning (reset) of session variable. The content of session variable should be cleaned when related session variable is removed from system catalog. But queue of sinval messages can be truncated, or current transaction state can disallow an access to system catalog, so cleaning can be postponed. --- src/backend/access/transam/xact.c | 2 +- src/backend/catalog/dependency.c | 5 + src/backend/catalog/namespace.c | 327 ++++++ src/backend/commands/meson.build | 1 + src/backend/commands/session_variable.c | 1235 +++++++++++++++++++-- src/backend/executor/execExpr.c | 76 ++ src/backend/executor/execExprInterp.c | 17 + src/backend/executor/execMain.c | 58 + src/backend/executor/execParallel.c | 147 ++- src/backend/jit/llvm/llvmjit_expr.c | 6 + src/backend/optimizer/plan/planner.c | 8 + src/backend/optimizer/plan/setrefs.c | 118 +- src/backend/optimizer/prep/prepjointree.c | 3 + src/backend/optimizer/util/clauses.c | 74 +- src/backend/parser/analyze.c | 7 + src/backend/parser/parse_expr.c | 253 ++++- src/backend/tcop/pquery.c | 3 + src/backend/utils/adt/ruleutils.c | 46 + src/backend/utils/cache/plancache.c | 29 +- src/backend/utils/fmgr/fmgr.c | 10 +- src/backend/utils/misc/guc_tables.c | 10 + src/include/catalog/namespace.h | 1 + src/include/catalog/pg_proc.dat | 9 +- src/include/commands/session_variable.h | 12 +- src/include/executor/execExpr.h | 11 + src/include/executor/execdesc.h | 4 + src/include/nodes/execnodes.h | 19 + src/include/nodes/parsenodes.h | 2 + src/include/nodes/pathnodes.h | 5 + src/include/nodes/plannodes.h | 2 + src/include/nodes/primnodes.h | 12 +- src/include/optimizer/planmain.h | 2 + src/include/parser/parse_expr.h | 1 + src/include/parser/parse_node.h | 1 + src/tools/pgindent/typedefs.list | 3 + 35 files changed, 2417 insertions(+), 102 deletions(-) diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index ee20c2e520..0ccdb56ff4 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -2225,7 +2225,7 @@ CommitTransaction(void) */ smgrDoPendingSyncs(true, is_parallel_worker); - /* Let ON COMMIT DROP */ + /* Let ON COMMIT DROP or ON TRANSACTION END */ AtPreEOXact_SessionVariable(true); /* close large objects before lower-level cleanup */ diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index 77acbeda80..cae1c7c1f0 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -1915,6 +1915,11 @@ find_expr_references_walker(Node *node, { Param *param = (Param *) node; + /* A variable parameter depends on the session variable */ + if (param->paramkind == PARAM_VARIABLE) + add_object_address(OCLASS_VARIABLE, param->paramvarid, 0, + context->addrs); + /* A parameter must depend on the parameter's datatype */ add_object_address(OCLASS_TYPE, param->paramtype, 0, context->addrs); diff --git a/src/backend/catalog/namespace.c b/src/backend/catalog/namespace.c index 9a04530d1e..73d7996160 100644 --- a/src/backend/catalog/namespace.c +++ b/src/backend/catalog/namespace.c @@ -2970,6 +2970,333 @@ LookupVariable(const char *nspname, return varoid; } +/* + * The input list contains names with indirection expressions used as the left + * part of LET statement. The following routine returns a new list with only + * initial strings (names) - without indirection expressions. + */ +List * +NamesFromList(List *names) +{ + ListCell *l; + List *result = NIL; + + foreach(l, names) + { + Node *n = lfirst(l); + + if (IsA(n, String)) + { + result = lappend(result, n); + } + else + break; + } + + return result; +} + +/* + * IdentifyVariable - try to find variable identified by list of names. + * + * Before this call we don't know, how these fields should be mapped to + * schema name, variable name and attribute name. In this routine + * we try to apply passed names to all possible combinations of schema name, + * variable name and attribute name, and we count valid combinations. + * + * Returns oid of identified variable. When last field of names list is + * identified as an attribute, then output attrname argument is set to + * an string of this field. + * + * When there is not any valid combination, then we are sure, so the + * list of names cannot to identify any session variable. In this case + * we return InvalidOid. + * + * We can find more valid combination than one. + * Example: users can have session variable x in schema y, and + * session variable y with attribute x inside some schema from + * search path. In this situation the meaning of expression "y"."x" + * is ambiguous. In this case this routine returns oid of variable + * x in schema y, and the output parameter "not_unique" is set to + * true. In this case this variable is locked. + * + * The AccessShareLock is created on related session variable. The lock + * will be kept for the whole transaction. + * + * Note: the out attrname should be used only when the session variable + * is identified. When the session variable is not identified, then this + * output variable can hold reference to some string, but isn't sure + * about its semantics. + * + * When we use this routine for identification of shadowed variable, + * we don't want to raise any error. Shadowing column reference is correct, + * and we don't want to break execution due shadowing check. + */ +Oid +IdentifyVariable(List *names, char **attrname, bool *not_unique, bool noerror) +{ + Node *field1 = NULL; + Node *field2 = NULL; + Node *field3 = NULL; + Node *field4 = NULL; + char *a = NULL; + char *b = NULL; + char *c = NULL; + char *d = NULL; + Oid varid = InvalidOid; + Oid old_varid = InvalidOid; + uint64 inval_count; + bool retry = false; + + /* + * DDL operations can change the results of a name lookup. Since all such + * operations will generate invalidation messages, we keep track of + * whether any such messages show up while we're performing the operation, + * and retry until either (1) no more invalidation messages show up or (2) + * the answer doesn't change. + */ + for (;;) + { + Oid varoid_without_attr = InvalidOid; + Oid varoid_with_attr = InvalidOid; + + *not_unique = false; + *attrname = NULL; + varid = InvalidOid; + + inval_count = SharedInvalidMessageCounter; + + switch (list_length(names)) + { + case 1: + field1 = linitial(names); + + Assert(IsA(field1, String)); + + varid = LookupVariable(NULL, strVal(field1), true); + break; + + case 2: + field1 = linitial(names); + field2 = lsecond(names); + + Assert(IsA(field1, String)); + a = strVal(field1); + + if (IsA(field2, String)) + { + /* when both fields are of string type */ + b = strVal(field2); + + /* + * a.b can mean "schema"."variable" or "variable"."field". + * Check both variants, and returns InvalidOid with + * not_unique flag, when both interpretations are + * possible. + */ + varoid_without_attr = LookupVariable(a, b, true); + varoid_with_attr = LookupVariable(NULL, a, true); + } + else + { + /* The last field of list can be star too. */ + Assert(IsA(field2, A_Star)); + + /* + * In this case, the field1 should be variable name. + * But direct unboxing of composite session variables + * is not supported now, and then we don't need to try + * lookup related variable. + * + * Unboxing is supported by syntax (var).* + */ + return InvalidOid; + } + + if (OidIsValid(varoid_without_attr) && OidIsValid(varoid_with_attr)) + { + *not_unique = true; + varid = varoid_without_attr; + } + else if (OidIsValid(varoid_without_attr)) + { + varid = varoid_without_attr; + } + else if (OidIsValid(varoid_with_attr)) + { + *attrname = b; + varid = varoid_with_attr; + } + break; + + case 3: + { + bool field1_is_catalog = false; + + field1 = linitial(names); + field2 = lsecond(names); + field3 = lthird(names); + + Assert(IsA(field1, String)); + Assert(IsA(field2, String)); + + a = strVal(field1); + b = strVal(field2); + + if (IsA(field3, String)) + { + c = strVal(field3); + + /* + * a.b.c can mean catalog.schema.variable + * or schema.variable.field. + * + * Check both variants, and set not_unique flag, + * when both interpretations are possible. + * + * When third node is star, only possible + * interpretation is schema.variable.*, but this + * pattern is not supported now. + */ + varoid_with_attr = LookupVariable(a, b, true); + + /* + * check pattern catalog.schema.variable only when + * there is possibility to success. + */ + if (strcmp(a, get_database_name(MyDatabaseId)) == 0) + { + field1_is_catalog = true; + varoid_without_attr = LookupVariable(b, c, true); + } + } + else + { + Assert(IsA(field3, A_Star)); + return InvalidOid; + } + + if (OidIsValid(varoid_without_attr) && OidIsValid(varoid_with_attr)) + { + *not_unique = true; + varid = varoid_without_attr; + } + else if (OidIsValid(varoid_without_attr)) + { + varid = varoid_without_attr; + } + else if (OidIsValid(varoid_with_attr)) + { + *attrname = c; + varid = varoid_with_attr; + } + + /* + * When we didn't find variable, we can (when it is allowed) + * raise cross-database reference error. + */ + if (!OidIsValid(varid) && !noerror && !field1_is_catalog) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cross-database references are not implemented: %s", + NameListToString(names)))); + } + break; + + case 4: + { + field1 = linitial(names); + field2 = lsecond(names); + field3 = lthird(names); + field4 = lfourth(names); + + Assert(IsA(field1, String)); + Assert(IsA(field2, String)); + Assert(IsA(field3, String)); + + a = strVal(field1); + b = strVal(field2); + c = strVal(field3); + + /* + * In this case, "a" is used as catalog name - check it. + */ + if (strcmp(a, get_database_name(MyDatabaseId)) != 0) + { + if (!noerror) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cross-database references are not implemented: %s", + NameListToString(names)))); + } + else + { + if (IsA(field4, String)) + { + d = strVal(field4); + } + else + { + Assert(IsA(field4, A_Star)); + return InvalidOid; + } + + *attrname = d; + varid = LookupVariable(b, c, true); + } + } + break; + + default: + if (!noerror) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("improper qualified name (too many dotted names): %s", + NameListToString(names)))); + return InvalidOid; + } + + /* + * If, upon retry, we get back the same OID we did last time, then the + * invalidation messages we processed did not change the final answer. + * So we're done. + * + * If we got a different OID, we've locked the variable that used to + * have this name rather than the one that does now. So release the + * lock. + */ + if (retry) + { + if (old_varid == varid) + break; + + if (OidIsValid(old_varid)) + UnlockDatabaseObject(VariableRelationId, old_varid, 0, AccessShareLock); + } + + /* + * Lock the variable. This will also accept any pending invalidation + * messages. If we got back InvalidOid, indicating not found, then + * there's nothing to lock, but we accept invalidation messages + * anyway, to flush any negative catcache entries that may be + * lingering. + */ + if (!OidIsValid(varid)) + AcceptInvalidationMessages(); + else if (OidIsValid(varid)) + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); + + if (inval_count == SharedInvalidMessageCounter) + break; + + retry = true; + old_varid = varid; + varid = InvalidOid; + } + + return varid; +} + /* * DeconstructQualifiedName * Given a possibly-qualified name expressed as a list of String nodes, diff --git a/src/backend/commands/meson.build b/src/backend/commands/meson.build index 42cced9ebe..404be3d22b 100644 --- a/src/backend/commands/meson.build +++ b/src/backend/commands/meson.build @@ -38,6 +38,7 @@ backend_sources += files( 'schemacmds.c', 'seclabel.c', 'sequence.c', + 'session_variable.c', 'statscmds.c', 'subscriptioncmds.c', 'tablecmds.c', diff --git a/src/backend/commands/session_variable.c b/src/backend/commands/session_variable.c index 946e73e467..b4a91ea5a3 100644 --- a/src/backend/commands/session_variable.c +++ b/src/backend/commands/session_variable.c @@ -8,7 +8,7 @@ * * * IDENTIFICATION - * src/backend/commands/sessionvariable.c + * src/backend/commands/session_variable.c * *------------------------------------------------------------------------- */ @@ -22,87 +22,244 @@ #include "commands/session_variable.h" #include "funcapi.h" #include "miscadmin.h" +#include "optimizer/optimizer.h" +#include "storage/lmgr.h" +#include "storage/proc.h" #include "utils/builtins.h" +#include "utils/datum.h" +#include "utils/inval.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/syscache.h" /* - * The life cycle of temporary session variable can be - * limmited by using clause ON COMMIT DROP. + * Values of session variables are stored in the backend local memory, + * in sessionvars hash table in binary format, in a dedicated memory + * context SVariableMemoryContext. A session variable value can stay + * valid for longer than the transaction that assigns its value. To + * make sure that the underlying memory is eventually freed, but not + * before it's guarantee that the value won't be needed anymore, we + * need to handle the two following points: + * + * - We need detect when a variable is dropped, whether in the current + * transaction in the current session or by another session, and mark + * the underlying entries for removal. To protect the content against + * possibly rollbacked DROP VARIABLE commands, the entries (and + * memory) shouldn't be freed immediately but be postponed until the + * end of the transaction. + * + * - The session variable can be dropped explicitly (by DROP VARIABLE + * command) or implicitly (using ON COMMIT DROP clause), and the + * value can be implicitly removed (using the ON TRANSACTION END + * clause). In all those cases the memory should also be freed at + * the transaction end. + * + * To achieve that, we maintain 3 queues of actions to be performed at + * certain time: + * - a List of SVariableXActActionItem, to handle ON COMMIT DROP + * variables, and delayed memory cleanup of variable dropped by the + * current transaction. Those actions are transactional (for instance + * we don't want to cleanup the memory of a rollbacked DROP VARIABLE) + * so the structure is needed to keep track of the final transaction + * state + * - a List of variable Oid for the ON TRANSACTION ON RESET variables + * - a List of variable Oid for the concurrent DROP VARIABLE + * notification we receive via shared invalidations. + * + * Note that although resetting a variable doesn't technically require + * to remove the entry from the sessionvars hash table, we currently + * do it. It's a simple way to implement the reset, and helps to + * reduce memory usage and prevents the hash table from bloating. + * + * There are two different ways to do the final access to session + * variables: buffered (indirect) or direct. Buffered access is used + * in regular DML statements, where we have to ensure the stability of + * the variable values. The session variables have the same behaviour + * as external query parameters, which is consistent with using + * PL/pgSQL's variables in embedded queries in PL/pgSQL. + * + * This is implemented by using an aux buffer (an array) that holds a + * copy of values of used (in query) session variables, which is also + * transmitted to the parallel workers. The values from this array + * are passed as constant (EEOP_CONST). + * + * Direct access is used by simple expression evaluation (PLpgSQL). + * In this case we don't need to ensure the stability of passed + * values, and maintaining the buffer with copies of values of session + * variables would be useless overhead. In this case we just read the + * value of the session variable directly (EEOP_PARAM_VARIABLE). This + * strategy removes the necessity to modify related PL/pgSQL code to + * support session variables (the reading of session variables is + * fully transparent for PL/pgSQL). */ typedef enum SVariableXActAction { SVAR_ON_COMMIT_DROP, /* used for ON COMMIT DROP */ + SVAR_ON_COMMIT_RESET, /* used for DROP VARIABLE */ } SVariableXActAction; typedef struct SVariableXActActionItem { Oid varid; /* varid of session variable */ + SVariableXActAction action; /* - * creating_subid is the ID of the creating subxact. If the action was - * unregistered during the current transaction, deleting_subid is the ID - * of the deleting subxact, otherwise InvalidSubTransactionId. + * creating_subid is the ID of the sub-transaction that registered + * the action. If the action was unregistered during the current + * transaction, deleting_subid is the ID of the deleting + * sub-transaction, otherwise InvalidSubTransactionId. */ SubTransactionId creating_subid; SubTransactionId deleting_subid; } SVariableXActActionItem; -/* List holds fields of SVariableXActActionItem type */ -static List *xact_drop_actions = NIL; - -static void register_session_variable_xact_action(Oid varid, SVariableXActAction action); -static void unregister_session_variable_xact_action(Oid varid, SVariableXActAction action); +/* List of SVariableXActActionItem */ +static List *xact_on_commit_actions = NIL; +/* + * To process ON TRANSACTION END RESET variables, for which we always + * need to clear the saved values. + */ +static List *xact_reset_varids = NIL; /* - * Do the necessary work to setup local memory management of a new - * variable. - * - * Caller should already have created the necessary entry in catalog - * and made them visible. + * When the session variable is dropped we need to free local memory. The + * session variable can be dropped by current session, but it can be + * dropped by other's sessions too, so we have to watch sinval message. + * But because we don't want to free local memory immediately, we need to + * hold list of possibly dropped session variables and at the end of + * transaction, we check session variables from this list against system + * catalog. This check can be postponed into next transaction if + * current transactions is in aborted state, as we wouldn't be able to + * access the system catalog. */ -void -SessionVariableCreatePostprocess(Oid varid, char eoxaction) +static List *xact_recheck_varids = NIL; + +typedef struct SVariableData { + Oid varid; /* pg_variable OID of the variable (hash key) */ + /* - * For temporary variables, we need to create a new end of xact action to - * ensure deletion from catalog. + * The session variable is identified by oid. The oid is unique in + * catalog. Unfortunately, the memory cleanup can be postponed to + * the beginning of the session next transaction, and it's possible + * that this next transaction sees a different variable with the same oid. + * We therefore need an extra identifier to distinguish both cases. We + * use the LSN number of session variable at creation time. The + * value of session variable (in memory) is valid, when there is a + * record in pg_variable with same oid and same create_lsn. */ - if (eoxaction == VARIABLE_EOX_DROP) - { - Assert(isTempNamespace(get_session_variable_namespace(varid))); + XLogRecPtr create_lsn; - register_session_variable_xact_action(varid, SVAR_ON_COMMIT_DROP); + bool isnull; + bool freeval; + Datum value; + + Oid typid; + int16 typlen; + bool typbyval; + + bool is_domain; + void *domain_check_extra; + LocalTransactionId domain_check_extra_lxid; + + /* + * Top level local transaction id of the last transaction that dropped the + * variable if any. We need this information to avoid freeing memory for + * variable dropped by the local backend that may be eventually rollbacked. + */ + LocalTransactionId drop_lxid; + + bool is_not_null; /* don't allow null values */ + bool is_immutable; /* true when variable is immutable */ + bool has_defexpr; /* true when variable has a default value */ + + bool is_valid; /* true when variable was successfully + * initialized */ + + uint32 hashvalue; /* used for pairing sinval message */ + + bool eox_reset; /* true, when lifecycle is limitted by + * transaction */ +} SVariableData; + +typedef SVariableData *SVariable; + +static HTAB *sessionvars = NULL; /* hash table for session variables */ + +static MemoryContext SVariableMemoryContext = NULL; + +static void create_sessionvars_hashtables(void); +static void free_session_variable_value(SVariable svar); +static void init_session_variable(SVariable svar, Variable *var); +static bool is_session_variable_valid(SVariable svar); +static void pg_variable_cache_callback(Datum arg, int cacheid, + uint32 hashvalue); +static SVariable prepare_variable_for_reading(Oid varid); +static void register_session_variable_xact_action(Oid varid, + SVariableXActAction action); +static void remove_session_variable(SVariable svar); +static void remove_session_variable_by_id(Oid varid); +static void set_session_variable(SVariable svar, Datum value, bool isnull, + bool init_mode); +static const char *SVariableXActActionName(SVariableXActAction action); +static void sync_sessionvars_all(bool filter_lxid); +static void unregister_session_variable_xact_action(Oid varid, + SVariableXActAction action); + + +/* + * Returns human readable name of SVariableXActAction value. + */ +static const char * +SVariableXActActionName(SVariableXActAction action) +{ + switch (action) + { + case SVAR_ON_COMMIT_DROP: + return "ON COMMIT DROP"; + case SVAR_ON_COMMIT_RESET: + return "ON COMMIT RESET"; + default: + elog(ERROR, "unknown SVariableXActAction action %d", action); } } /* - * Handle the local memory cleanup for a DROP VARIABLE command. - * - * Caller should take care of removing the pg_variable entry first. + * Free all memory allocated for the given session variable, but + * preserve the hash entry in sessionvars. */ -void -SessionVariableDropPostprocess(Oid varid) +static void +free_session_variable_value(SVariable svar) { + /* Clean current value */ + if (!svar->isnull) + { + if (svar->freeval) + { + pfree(DatumGetPointer(svar->value)); + svar->freeval = false; + } + + svar->isnull = true; + } + + svar->value = (Datum) 0; + svar->freeval = false; + /* - * The entry was removed from catalog already, we must not do it - * again at end of xact time. + * We can mark this session variable as valid when it has not default + * expression, and when null is allowed. When it has defexpr, then the + * content will be valid after an assignment or defexp evaluation. */ - unregister_session_variable_xact_action(varid, SVAR_ON_COMMIT_DROP); + svar->is_valid = !svar->has_defexpr && !svar->is_not_null; } /* * Registration of actions to be executed on session variables at transaction * end time. We want to drop temporary session variables with clause ON COMMIT - * DROP, or we want to reset values of session variables with clause ON - * TRANSACTION END RESET or we want to clean (reset) local memory allocated by - * values of dropped session variables. - */ - -/* - * Register a session variable xact action. + * DROP, or we want to clean (reset) local memory allocated by + * values of session variables dropped in other backends. */ static void register_session_variable_xact_action(Oid varid, @@ -111,27 +268,30 @@ register_session_variable_xact_action(Oid varid, SVariableXActActionItem *xact_ai; MemoryContext oldcxt; - oldcxt = MemoryContextSwitchTo(CacheMemoryContext); + elog(DEBUG1, "SVariableXActAction \"%s\" is registered for session variable (oid:%u)", + SVariableXActActionName(action), varid); + + oldcxt = MemoryContextSwitchTo(TopTransactionContext); xact_ai = (SVariableXActActionItem *) palloc(sizeof(SVariableXActActionItem)); xact_ai->varid = varid; + xact_ai->action = action; xact_ai->creating_subid = GetCurrentSubTransactionId(); xact_ai->deleting_subid = InvalidSubTransactionId; - Assert(action == SVAR_ON_COMMIT_DROP); - xact_drop_actions = lcons(xact_ai, xact_drop_actions); + xact_on_commit_actions = lcons(xact_ai, xact_on_commit_actions); MemoryContextSwitchTo(oldcxt); } /* - * Unregister an action on a given session variable from action list. In this - * moment, the action is just marked as deleted by setting deleting_subid. The - * calling even might be rollbacked, in which case we should not lose this - * action. + * Unregister an action on a given session variable from the action list. + * The action is just marked as deleted by setting deleting_subid. + * The calling subtransaction even might be rollbacked, in which case the + * action shouldn't be removed. */ static void unregister_session_variable_xact_action(Oid varid, @@ -139,43 +299,842 @@ unregister_session_variable_xact_action(Oid varid, { ListCell *l; - Assert(action == SVAR_ON_COMMIT_DROP); + elog(DEBUG1, "SVariableXActAction \"%s\" is unregistered for session variable (oid:%u)", + SVariableXActActionName(action), varid); - foreach(l, xact_drop_actions) + foreach(l, xact_on_commit_actions) { SVariableXActActionItem *xact_ai = (SVariableXActActionItem *) lfirst(l); - if (xact_ai->varid == varid) + if (xact_ai->action == action && xact_ai->varid == varid) xact_ai->deleting_subid = GetCurrentSubTransactionId(); } } /* - * Perform ON TRANSACTION END RESET or ON COMMIT DROP - * and COMMIT/ROLLBACK of transaction session variables. + * Release the given session variable from sessionvars hashtab and free + * all underlying allocated memory. + */ +static void +remove_session_variable(SVariable svar) +{ + free_session_variable_value(svar); + + /* + * In this moment, the session variable is not in catalog, so only saved + * oid can be displayed. + */ + elog(DEBUG1, "session variable (oid:%u) is removing from memory", + svar->varid); + + if (hash_search(sessionvars, + (void *) &svar->varid, + HASH_REMOVE, + NULL) == NULL) + elog(DEBUG1, "hash table corrupted"); +} + +/* + * Release the session variable defined by varid from sessionvars + * hashtab and free all underlying allocated memory. + */ +static void +remove_session_variable_by_id(Oid varid) +{ + SVariable svar; + bool found; + + if (!sessionvars) + return; + + svar = (SVariable) hash_search(sessionvars, &varid, + HASH_FIND, &found); + if (found) + remove_session_variable(svar); +} + +/* + * Callback function for session variable invalidation. + * + * It queues a list of variable Oid in xact_recheck_varids. + */ +static void +pg_variable_cache_callback(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS status; + SVariable svar; + + /* + * There is no guarantee of sessionvars being initialized, even when + * receiving an invalidation callback, as DISCARD [ ALL | VARIABLES ] + * destroys the hash table entirely. + */ + if (!sessionvars) + return; + + elog(DEBUG1, "pg_variable_cache_callback %u %u", cacheid, hashvalue); + + /* + * When the hashvalue is not specified, then we have to recheck all + * currently used session variables. Since we can't guarantee the exact + * session variable from its hashValue, we also have to iterate over + * all items of the sessionvars hash table. + */ + hash_seq_init(&status, sessionvars); + + while ((svar = (SVariable) hash_seq_search(&status)) != NULL) + { + if (hashvalue == 0 || svar->hashvalue == hashvalue) + { + MemoryContext oldcxt; + + /* The list needs to be able to survive the transaction */ + oldcxt = MemoryContextSwitchTo(SVariableMemoryContext); + + xact_recheck_varids = lappend_oid(xact_recheck_varids, + svar->varid); + + MemoryContextSwitchTo(oldcxt); + + elog(DEBUG1, "session variable (oid:%u) should be rechecked (forced by sinval)", + svar->varid); + } + + /* + * although it there is low probability, we have to iterate over all + * locally set session variables, because hashvalue is not a unique + * identifier. + */ + } +} + +/* + * Returns true when the entry in pg_variable is valid for the given session + * variable. + */ +static bool +is_session_variable_valid(SVariable svar) +{ + HeapTuple tp; + bool result = false; + + tp = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(svar->varid)); + + if (HeapTupleIsValid(tp)) + { + /* + * In this case, the only oid cannot be used as unique identifier, + * because the oid counter can wraparound, and the oid can be used for + * new other session variable. We do a second check against 64bit + * unique identifier. + */ + if (svar->create_lsn == ((Form_pg_variable) GETSTRUCT(tp))->varcreate_lsn) + result = true; + + ReleaseSysCache(tp); + } + + return result; +} + +/* + * Recheck the possibly invalidated variables (in memory) against system + * catalog. This routine is called before any read or any write from/to session + * variables and when processing a committed transaction. + * If filter_lxid is true, this function will ignore the recheck for variables + * that have the same cached local transaction id as the transaction current + * top level local transaction id, ie. the variables dropped in the current top + * level transaction or any underlying subtransaction. + */ +static void +sync_sessionvars_all(bool filter_lxid) +{ + SVariable svar; + ListCell *l; + List *xact_recheck_varids_snapshot; + + if (!xact_recheck_varids) + return; + + /* + * If the sessionvars hashtable is NULL (which can be done by DISCARD + * VARIABLES), we are sure that there aren't any active session variable + * in this session. + */ + if (!sessionvars) + { + list_free(xact_recheck_varids); + xact_recheck_varids = NIL; + return; + } + + elog(DEBUG1, "effective call of sync_sessionvars_all()"); + + /* + * The recheck list can contain many duplicates, so clean it up before + * processing to avoid extraneous work. + */ + list_sort(xact_recheck_varids, list_oid_cmp); + list_deduplicate_oid(xact_recheck_varids); + + /* + * When we check the variables, the system cache can be invalidated, + * and then xact_recheck_varids can be extended. But we need to iterate + * over stable list, and we must not at same time discard invalidation + * messages. + * + * Steps of possible solution: + * + * 1. move xact_recheck_varids to aux variable, and reset + * xact_recheck_varids, + * + * 2. process fields in list of aux variable, + * + * 3. merge the content of aux variable back to xact_recheck_varids. + */ + xact_recheck_varids_snapshot = xact_recheck_varids; + xact_recheck_varids = NIL; + + /* + * This routine is called before any reading, so the session should be in + * transaction state. This is required to access the system catalog. + */ + Assert(IsTransactionState()); + + foreach(l, xact_recheck_varids_snapshot) + { + bool found; + Oid varid = lfirst_oid(l); + + svar = (SVariable) hash_search(sessionvars, &varid, + HASH_FIND, &found); + + /* + * Remove invalid variables, but don't touch variables that were + * dropped by the current top level local transaction or any + * subtransaction underneath, as there's no guarantee that the + * transaction will be committed. Such variables will be removed in + * the next transaction if needed. + */ + if (found) + { + /* + * If this is a variable dropped by the current transaction, + * ignore it and keep the oid to recheck in the next transaction. + */ + if (filter_lxid && svar->drop_lxid == MyProc->lxid) + continue; + + if (!is_session_variable_valid(svar)) + remove_session_variable(svar); + } + + /* + * If caller asked to filter the list, we have to clean items as they + * are processed. + */ + if (filter_lxid) + xact_recheck_varids_snapshot = + foreach_delete_current(xact_recheck_varids_snapshot, + l); + } + + /* + * If caller ask to filter the list, some items are not processed + * and we should to merge these items to xact_recheck_varids. + */ + if (filter_lxid) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(SVariableMemoryContext); + + xact_recheck_varids = list_concat(xact_recheck_varids, + xact_recheck_varids_snapshot); + + MemoryContextSwitchTo(oldcxt); + } + + list_free(xact_recheck_varids_snapshot); +} + +/* + * Create the hash table for storing session variables. + */ +static void +create_sessionvars_hashtables(void) +{ + HASHCTL vars_ctl; + + Assert(!sessionvars); + + /* set callbacks */ + if (!SVariableMemoryContext) + { + /* Read sinval messages */ + CacheRegisterSyscacheCallback(VARIABLEOID, + pg_variable_cache_callback, + (Datum) 0); + + /* We need our own long lived memory context */ + SVariableMemoryContext = + AllocSetContextCreate(TopMemoryContext, + "session variables", + ALLOCSET_START_SMALL_SIZES); + } + + Assert(SVariableMemoryContext); + + memset(&vars_ctl, 0, sizeof(vars_ctl)); + vars_ctl.keysize = sizeof(Oid); + vars_ctl.entrysize = sizeof(SVariableData); + vars_ctl.hcxt = SVariableMemoryContext; + + Assert(sessionvars == NULL); + + sessionvars = hash_create("Session variables", 64, &vars_ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); +} + +/* + * Assign some content to the session variable. It's copied to + * SVariableMemoryContext if necessary. + * + * init_mode is true when the value of session variable should be initialized + * by the default expression if any. This is the only case where we allow the + * modification of an immutable variables with default expression. + * + * If any error happens, the existing value shouldn't be modified. + */ +static void +set_session_variable(SVariable svar, Datum value, bool isnull, bool init_mode) +{ + Datum newval = value; + + Assert(svar && OidIsValid(svar->typid)); + + /* Don't allow assignment of null to NOT NULL variable */ + if (isnull && svar->is_not_null) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("null value is not allowed for NOT NULL session variable \"%s.%s\"", + get_namespace_name(get_session_variable_namespace(svar->varid)), + get_session_variable_name(svar->varid)))); + + /* + * Don't allow the modification of an immutable session variable that + * already has an assigned value (possibly NULL) or has a default + * expression (in which case the value should always be the result of + * default expression evaluation) unless the variable is being initialized. + */ + if (!init_mode && + (svar->is_immutable && + (svar->is_valid || svar->has_defexpr))) + ereport(ERROR, + (errcode(ERRCODE_ERROR_IN_ASSIGNMENT), + errmsg("session variable \"%s.%s\" is declared IMMUTABLE", + get_namespace_name(get_session_variable_namespace(svar->varid)), + get_session_variable_name(svar->varid)))); + + if (!isnull) + { + MemoryContext oldcxt = MemoryContextSwitchTo(SVariableMemoryContext); + + newval = datumCopy(value, svar->typbyval, svar->typlen); + + MemoryContextSwitchTo(oldcxt); + } + else + { + /* The caller shouldn't have provided any real value. */ + Assert(value == (Datum) 0); + } + + free_session_variable_value(svar); + + svar->value = newval; + + svar->isnull = isnull; + svar->freeval = newval != value; + svar->is_valid = true; + + /* + * XXX While unlikely, an error here is possible. + * It wouldn't leak memory as the allocated chunk has already been + * correctly assigned to the session variable, but would contradict this + * function contract, which is that this function should either succeed or + * leave the current value untouched. + */ + elog(DEBUG1, "session variable \"%s.%s\" (oid:%u) has new value", + get_namespace_name(get_session_variable_namespace(svar->varid)), + get_session_variable_name(svar->varid), + svar->varid); +} + +/* + * Initialize session variable svar from variable var + */ +static void +init_session_variable(SVariable svar, Variable *var) +{ + MemoryContext oldcxt; + + Assert(OidIsValid(var->oid)); + + svar->varid = var->oid; + svar->create_lsn = var->create_lsn; + + svar->isnull = true; + svar->freeval = false; + svar->value = (Datum) 0; + + svar->typid = var->typid; + get_typlenbyval(var->typid, &svar->typlen, &svar->typbyval); + + svar->is_domain = (get_typtype(var->typid) == TYPTYPE_DOMAIN); + svar->domain_check_extra = NULL; + svar->domain_check_extra_lxid = InvalidLocalTransactionId; + + svar->drop_lxid = InvalidLocalTransactionId; + + svar->is_not_null = var->is_not_null; + svar->is_immutable = var->is_immutable; + svar->has_defexpr = var->has_defexpr; + + /* the value of variable is not known yet */ + svar->is_valid = false; + + svar->hashvalue = GetSysCacheHashValue1(VARIABLEOID, + ObjectIdGetDatum(var->oid)); + + svar->eox_reset = var->eoxaction == VARIABLE_EOX_RESET || + var->eoxaction == VARIABLE_EOX_DROP; + + oldcxt = MemoryContextSwitchTo(TopTransactionContext); + + if (svar->eox_reset) + xact_reset_varids = lappend_oid(xact_reset_varids, var->oid); + + MemoryContextSwitchTo(oldcxt); +} + +/* + * Search a seesion variable in the hash table given its oid. If it + * doesn't exist, then insert it (and calculate defexpr if it exists). + * + * Caller is responsible for doing permission checks. + * + * As side effect this function acquires AccessShareLock on + * related session variable until the end of the transaction. + */ +static SVariable +prepare_variable_for_reading(Oid varid) +{ + SVariable svar; + Variable var; + bool found; + + var.oid = InvalidOid; + + if (!sessionvars) + create_sessionvars_hashtables(); + + /* Protect used session variable against drop until transaction end */ + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); + + /* + * Make sure that all entries in sessionvars hash table are valid, but + * keeping variables dropped by the current transaction. + */ + sync_sessionvars_all(true); + + svar = (SVariable) hash_search(sessionvars, &varid, + HASH_ENTER, &found); + + /* Return content if it is available and valid */ + if (!found || !svar->is_valid) + { + /* We need to load defexpr. */ + InitVariable(&var, varid, false); + + if (!found) + { + init_session_variable(svar, &var); + + elog(DEBUG1, "session variable \"%s.%s\" (oid:%u) has new entry in memory (emitted by READ)", + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid), + varid); + } + + /* + * Raise an error if this is a NOT NULL variable without default + * expression. + */ + if (var.is_not_null && !var.defexpr) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("null value is not allowed for NOT NULL session variable \"%s.%s\"", + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)), + errdetail("The session variable was not initialized yet."))); + + if (svar->has_defexpr) + { + Datum value = (Datum) 0; + bool isnull; + EState *estate = NULL; + Expr *defexpr; + ExprState *defexprs; + MemoryContext oldcxt; + + /* Prepare default expr */ + estate = CreateExecutorState(); + + oldcxt = MemoryContextSwitchTo(estate->es_query_cxt); + + defexpr = expression_planner((Expr *) var.defexpr); + defexprs = ExecInitExpr(defexpr, NULL); + value = ExecEvalExprSwitchContext(defexprs, + GetPerTupleExprContext(estate), + &isnull); + + MemoryContextSwitchTo(oldcxt); + + /* Store result before releasing Executor memory */ + set_session_variable(svar, value, isnull, true); + + FreeExecutorState(estate); + } + else + set_session_variable(svar, (Datum) 0, true, true); + } + + /* + * Although the value of domain type should be valid (it is checked when + * it is assigned to session variable), we have to check related + * constraints each time we access the variable. It can be more expensive + * than in PL/pgSQL, as PL/pgSQL forces domain checks only when the value is assigned + * to the variable or when the value is returned from function. + * However, domain types have a constraint cache so it's not too much + * expensive.. + */ + if (svar->is_domain) + { + /* + * Store domain_check extra in TopTransactionContext. When we are in + * other transaction, the domain_check_extra cache is not valid + * anymore. + */ + if (svar->domain_check_extra_lxid != MyProc->lxid) + svar->domain_check_extra = NULL; + + domain_check(svar->value, svar->isnull, + svar->typid, &svar->domain_check_extra, + TopTransactionContext); + + svar->domain_check_extra_lxid = MyProc->lxid; + } + + return svar; +} + +/* + * Store the given value in an SVariable, and cache it if not already present. + * + * Caller is responsible for doing permission checks. + * + * As side effect this function acquires AccessShareLock on + * related session variable until the end of the transaction. + */ +void +SetSessionVariable(Oid varid, Datum value, bool isNull) +{ + SVariable svar; + bool found; + + if (!sessionvars) + create_sessionvars_hashtables(); + + /* Protect used session variable against drop until transaction end */ + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); + + /* + * Make sure that all entries in sessionvars hash table are valid, but + * keeping variables dropped by the current transaction. + */ + sync_sessionvars_all(true); + + svar = (SVariable) hash_search(sessionvars, &varid, + HASH_ENTER, &found); + + if (!found) + { + Variable var; + + /* We don't need to know defexpr here */ + InitVariable(&var, varid, true); + init_session_variable(svar, &var); + + elog(DEBUG1, "session variable \"%s.%s\" (oid:%u) has new entry in memory (emitted by WRITE)", + get_namespace_name(get_session_variable_namespace(svar->varid)), + get_session_variable_name(svar->varid), + varid); + } + + /* + * This should either succeed or fail without changing the currently stored + * value. + */ + set_session_variable(svar, value, isNull, false); +} + +/* + * Wrapper around SetSessionVariable after checking for correct permission. + */ +void +SetSessionVariableWithSecurityCheck(Oid varid, Datum value, bool isNull) +{ + AclResult aclresult; + + /* + * Is caller allowed to update the session variable? + */ + aclresult = object_aclcheck(VariableRelationId, varid, GetUserId(), ACL_UPDATE); + if (aclresult != ACLCHECK_OK) + aclcheck_error(aclresult, OBJECT_VARIABLE, get_session_variable_name(varid)); + + SetSessionVariable(varid, value, isNull); +} + +/* + * Returns a copy of the value of the session variable specified by its oid. + * Caller is responsible for doing permission checks. + */ +Datum +CopySessionVariable(Oid varid, bool *isNull, Oid *typid) +{ + SVariable svar; + Datum result; + + svar = prepare_variable_for_reading(varid); + Assert(svar != NULL && svar->is_valid); + + *typid = svar->typid; + + /* force copy of non NULL value */ + if (!svar->isnull) + { + result = datumCopy(svar->value, svar->typbyval, svar->typlen); + *isNull = false; + } + else + { + result = (Datum) 0; + *isNull = true; + } + + return (Datum) result; +} + +/* + * Returns a copy of ths value of the session variable specified by its oid + * with a check of the expected type. Like previous CopySessionVariable, the + * caller is responsible for doing permission checks. + */ +Datum +CopySessionVariableWithTypeCheck(Oid varid, bool *isNull, Oid expected_typid) +{ + SVariable svar; + Datum result; + + svar = prepare_variable_for_reading(varid); + Assert(svar != NULL && svar->is_valid); + + if (expected_typid != svar->typid) + elog(ERROR, "type of variable \"%s.%s\" is different than expected", + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)); + + if (!svar->isnull) + { + result = datumCopy(svar->value, svar->typbyval, svar->typlen); + *isNull = false; + } + else + { + result = (Datum) 0; + *isNull = true; + } + + return (Datum) result; +} + +/* + * Do the necessary work to setup local memory management of a new + * variable. + * + * Caller should already have created the necessary entry in catalog + * and made them visible. + */ +void +SessionVariableCreatePostprocess(Oid varid, char eoxaction) +{ + /* + * For temporary variables, we need to create a new end of xact action to + * ensure deletion from catalog. + */ + if (eoxaction == VARIABLE_EOX_DROP) + { + Assert(isTempNamespace(get_session_variable_namespace(varid))); + + register_session_variable_xact_action(varid, SVAR_ON_COMMIT_DROP); + } +} + +/* + * Handle the local memory cleanup for a DROP VARIABLE command. + * + * Caller should take care of removing the pg_variable entry first. + */ +void +SessionVariableDropPostprocess(Oid varid) +{ + /* + * The entry was removed from catalog already, we must not do it + * again at end of xact time. + */ + unregister_session_variable_xact_action(varid, SVAR_ON_COMMIT_DROP); + + if (sessionvars) + { + bool found; + SVariable svar = (SVariable) hash_search(sessionvars, &varid, + HASH_FIND, &found); + + if (found) + { + /* + * Save the current top level local transaction id to make sure we + * don't automatically remove the local variable storage in + * sync_sessionvars_all, as the DROP VARIABLE will send an + * invalidation message. + */ + Assert(LocalTransactionIdIsValid(MyProc->lxid)); + svar->drop_lxid = MyProc->lxid; + + /* + * For variables that are not ON TRANSACTION END RESET, we need to + * register an SVAR_ON_COMMIT_RESET action to free the local + * memory for this variable when the top level transaction + * is committed (we don't need to wait for sinval + * message). The cleanup action for one session variable can be + * duplicated in the action list without causing any problem, so we + * don't need to ensure uniqueness. We need a different action + * from RESET, because RESET is executed on any transaction end, + * but we want to execute this cleanup only when the current + * transaction will be committed. This action can be reverted by + * ABORT of DROP VARIABLE command. + */ + if (!svar->eox_reset) + register_session_variable_xact_action(varid, + SVAR_ON_COMMIT_RESET); + } + } +} + +/* + * Fast drop of the complete content of all session variables hash table, and + * cleanup of any list that wouldn't be relevant anymore. + * This is used by DISCARD VARIABLES (and DISCARD ALL) command. + */ +void +ResetSessionVariables(void) +{ + ListCell *lc; + + /* Destroy hash table and reset related memory context */ + if (sessionvars) + { + hash_destroy(sessionvars); + sessionvars = NULL; + } + + /* Release memory allocated by session variables */ + if (SVariableMemoryContext != NULL) + MemoryContextReset(SVariableMemoryContext); + + /* + * There isn't any session variable left, but we still need to retain the + * ON COMMIT DROP actions if any. + */ + foreach(lc, xact_on_commit_actions) + { + SVariableXActActionItem *xact_ai = + (SVariableXActActionItem *) lfirst(lc); + + if (xact_ai->action == SVAR_ON_COMMIT_DROP) + continue; + + pfree(xact_ai); + xact_on_commit_actions = foreach_delete_current(xact_on_commit_actions, + lc); + } + + /* We should clean xact_reset_varids */ + list_free(xact_reset_varids); + xact_reset_varids = NIL; + + /* + * xact_recheck_varids is stored in SVariableMemoryContext, so it has + * already been freed, just reset the list. + */ + xact_recheck_varids = NIL; +} + +/* + * Perform the necessary work for ON TRANSACTION END RESET and ON COMMIT DROP + * session variables. + * If the transaction is committed, also process the delayed memory cleanup of + * local DROP VARIABLE and process all pending rechecks. */ void AtPreEOXact_SessionVariable(bool isCommit) { ListCell *l; - foreach(l, xact_drop_actions) + /* + * Clean memory for all ON TRANSACTION END RESET variables. Do it first, + * as it reduces the overhead of the RECHECK action list. + */ + foreach(l, xact_reset_varids) { - SVariableXActActionItem *xact_ai = - (SVariableXActActionItem *) lfirst(l); + remove_session_variable_by_id(lfirst_oid(l)); + } - /* Iterate only over entries that are still pending */ - if (xact_ai->deleting_subid == InvalidSubTransactionId) + /* We can now clean xact_reset_varids */ + list_free(xact_reset_varids); + xact_reset_varids = NIL; + + if (isCommit && xact_on_commit_actions) + { + foreach(l, xact_on_commit_actions) { + SVariableXActActionItem *xact_ai = + (SVariableXActActionItem *) lfirst(l); + + /* Iterate only over entries that are still pending */ + if (xact_ai->deleting_subid != InvalidSubTransactionId) + continue; /* - * ON COMMIT DROP is allowed only for temp session variables. So - * we should explicitly delete only when current transaction was - * committed. When it's rollback, then session variable is removed - * automatically. + * ON COMMIT DROP is allowed only for temp session variables. + * So we should explicitly delete only when the current + * transaction is committed. When it's rollbacked, the session + * variable is removed automatically. */ - if (isCommit) + if (xact_ai->action == SVAR_ON_COMMIT_DROP) { ObjectAddress object; @@ -191,10 +1150,30 @@ AtPreEOXact_SessionVariable(bool isCommit) elog(DEBUG1, "session variable (oid:%u) will be deleted (forced by SVAR_ON_COMMIT_DROP action)", xact_ai->varid); + /* + * If the variable was locally set, the memory will be + * automatically cleaned up when we process the underlying + * shared invalidation for this drop. There can't be a recheck + * action for this variable, so there's nothing to gain + * explicitly removing it here. + */ performDeletion(&object, DROP_CASCADE, PERFORM_DELETION_INTERNAL | PERFORM_DELETION_QUIETLY); } + else + { + /* + * When we process DROP VARIABLE statement issued by the + * current transaction, we create an SVAR_ON_COMMIT_RESET xact + * action. We want to process this action only when related + * transaction is commited (when DROP VARIABLE statement + * sucessfully processed) as we need to preserve the variable + * content if the transaction that issued the DROP VARAIBLE + * statement is rollbacked. + */ + remove_session_variable_by_id(xact_ai->varid); + } } } @@ -202,12 +1181,26 @@ AtPreEOXact_SessionVariable(bool isCommit) * Any drop action left is an entry that was unregistered and not * rollbacked, so we can simply remove them. */ - list_free_deep(xact_drop_actions); - xact_drop_actions = NIL; + list_free_deep(xact_on_commit_actions); + xact_on_commit_actions = NIL; + + /* + * We process the list of recheck last for performance reason,the previous + * steps might remove entries from the hash table. + * We need catalog access to process the recheck, so this can only be done + * if the transaction is committed. Otherwise, we just keep the recheck + * list as-is and it will be processed at the next (committed) transaction. + */ + if (isCommit && xact_recheck_varids) + { + Assert(sessionvars); + + sync_sessionvars_all(false); + } } /* - * Post-subcommit or post-subabort cleanup of xact action list. + * Post-subcommit or post-subabort cleanup of xact_on_commit_actions list. * * During subabort, we can immediately remove entries created during this * subtransaction. During subcommit, just transfer entries marked during @@ -220,24 +1213,130 @@ AtEOSubXact_SessionVariable(bool isCommit, { ListCell *cur_item; - foreach(cur_item, xact_drop_actions) + foreach(cur_item, xact_on_commit_actions) { SVariableXActActionItem *xact_ai = (SVariableXActActionItem *) lfirst(cur_item); + /* + * The subtransaction that created this entry was rollbacked, we can + * remove it. + */ if (!isCommit && xact_ai->creating_subid == mySubid) { /* cur_item must be removed */ - xact_drop_actions = foreach_delete_current(xact_drop_actions, cur_item); + xact_on_commit_actions = foreach_delete_current(xact_on_commit_actions, cur_item); pfree(xact_ai); } else { - /* cur_item must be preserved */ + /* Otherwise cur_item must be preserved */ if (xact_ai->creating_subid == mySubid) xact_ai->creating_subid = parentSubid; if (xact_ai->deleting_subid == mySubid) - xact_ai->deleting_subid = isCommit ? parentSubid : InvalidSubTransactionId; + xact_ai->deleting_subid = isCommit ? parentSubid + : InvalidSubTransactionId; + } + } +} + +/* + * pg_session_variables - designed for testing + * + * This is a function designed for testing and debugging. It returns the + * content of sessionvars as-is, and can therefore display entries about + * session variables that were dropped but for which this backend didn't + * process the shared invalidations yet. + */ +Datum +pg_session_variables(PG_FUNCTION_ARGS) +{ +#define NUM_PG_SESSION_VARIABLES_ATTS 10 + + elog(DEBUG1, "pg_session_variables start"); + + InitMaterializedSRF(fcinfo, 0); + + if (sessionvars) + { + ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; + HASH_SEQ_STATUS status; + SVariable svar; + + /* + * Make sure that all entries in sessionvars hash table are valid, but + * keeping variables dropped by the current transaction. + */ + sync_sessionvars_all(true); + + hash_seq_init(&status, sessionvars); + + while ((svar = (SVariable) hash_seq_search(&status)) != NULL) + { + Datum values[NUM_PG_SESSION_VARIABLES_ATTS]; + bool nulls[NUM_PG_SESSION_VARIABLES_ATTS]; + HeapTuple tp; + + memset(values, 0, sizeof(values)); + memset(nulls, 0, sizeof(nulls)); + + values[0] = ObjectIdGetDatum(svar->varid); + values[3] = ObjectIdGetDatum(svar->typid); + + /* check if session variable is visible in system catalog */ + tp = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(svar->varid)); + + /* + * Sessionvars can hold data of variables removed from catalog, + * (and not purged) and then namespacename and name cannot be read + * from catalog. + */ + if (HeapTupleIsValid(tp)) + { + Form_pg_variable varform = (Form_pg_variable) GETSTRUCT(tp); + + /* When we see data in catalog */ + values[1] = CStringGetTextDatum( + get_namespace_name(varform->varnamespace)); + + values[2] = CStringGetTextDatum(NameStr(varform->varname)); + + values[4] = CStringGetTextDatum(format_type_be(svar->typid)); + values[5] = BoolGetDatum(false); + values[6] = BoolGetDatum(svar->is_valid); + + values[8] = BoolGetDatum( + object_aclcheck(VariableRelationId, svar->varid, + GetUserId(), ACL_SELECT) == ACLCHECK_OK); + + values[9] = BoolGetDatum( + object_aclcheck(VariableRelationId, svar->varid, + GetUserId(), ACL_UPDATE) == ACLCHECK_OK); + + ReleaseSysCache(tp); + } + else + { + /* + * When session variable was removed from catalog, but we + * haven't processed the invlidation yet. + */ + nulls[1] = true; + values[2] = CStringGetTextDatum( + DatumGetCString(DirectFunctionCall1(oidout, svar->varid))); + values[4] = PointerGetDatum( + cstring_to_text(format_type_be(svar->typid))); + values[5] = BoolGetDatum(true); + values[6] = BoolGetDatum(svar->is_valid); + nulls[7] = true; + nulls[8] = true; + } + + tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls); } } + + elog(DEBUG1, "pg_session_variables end"); + + return (Datum) 0; } diff --git a/src/backend/executor/execExpr.c b/src/backend/executor/execExpr.c index dcf56446c7..4136595c9f 100644 --- a/src/backend/executor/execExpr.c +++ b/src/backend/executor/execExpr.c @@ -34,6 +34,7 @@ #include "catalog/objectaccess.h" #include "catalog/pg_proc.h" #include "catalog/pg_type.h" +#include "commands/session_variable.h" #include "executor/execExpr.h" #include "executor/nodeSubplan.h" #include "funcapi.h" @@ -983,6 +984,81 @@ ExecInitExprRec(Expr *node, ExprState *state, scratch.d.param.paramtype = param->paramtype; ExprEvalPushStep(state, &scratch); break; + + case PARAM_VARIABLE: + { + int es_num_session_variables = 0; + SessionVariableValue *es_session_variables = NULL; + + if (state->parent && state->parent->state) + { + es_session_variables = state->parent->state->es_session_variables; + es_num_session_variables = state->parent->state->es_num_session_variables; + } + + if (es_session_variables) + { + SessionVariableValue *var; + + /* + * Use buffered session variables when the + * buffer with copied values is avaiable + * (standard query executor mode) + */ + + /* Parameter sanity checks. */ + if (param->paramid >= es_num_session_variables) + elog(ERROR, "paramid of PARAM_VARIABLE param is out of range"); + + var = &es_session_variables[param->paramid]; + + if (var->typid != param->paramtype) + elog(ERROR, "type of buffered value is different than PARAM_VARIABLE type"); + + /* + * In this case, pass the value like + * a constant. + */ + scratch.opcode = EEOP_CONST; + scratch.d.constval.value = var->value; + scratch.d.constval.isnull = var->isnull; + ExprEvalPushStep(state, &scratch); + } + else + { + AclResult aclresult; + Oid varid = param->paramvarid; + Oid vartype = param->paramtype; + + /* + * When the expression is evaluated directly + * without query executor start (plpgsql simple + * expr evaluation), then the array es_session_variables + * is null. In this case we need to use direct + * access to session variables. The values are + * not protected by using copy, but it is not + * problem (we don't need to emulate stability + * of the value). + * + * In this case we should to do aclcheck, because + * usual aclcheck from standard_ExecutorStart + * is not executed in this case. Fortunately + * it is just once per transaction. + */ + aclresult = object_aclcheck(VariableRelationId, varid, + GetUserId(), ACL_SELECT); + if (aclresult != ACLCHECK_OK) + aclcheck_error(aclresult, OBJECT_VARIABLE, + get_session_variable_name(varid)); + + scratch.opcode = EEOP_PARAM_VARIABLE; + scratch.d.vparam.varid = varid; + scratch.d.vparam.vartype = vartype; + ExprEvalPushStep(state, &scratch); + } + } + break; + case PARAM_EXTERN: /* diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c index ca44d39100..4a1d915269 100644 --- a/src/backend/executor/execExprInterp.c +++ b/src/backend/executor/execExprInterp.c @@ -59,6 +59,7 @@ #include "access/heaptoast.h" #include "catalog/pg_type.h" #include "commands/sequence.h" +#include "commands/session_variable.h" #include "executor/execExpr.h" #include "executor/nodeSubplan.h" #include "funcapi.h" @@ -449,6 +450,7 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull) &&CASE_EEOP_PARAM_EXEC, &&CASE_EEOP_PARAM_EXTERN, &&CASE_EEOP_PARAM_CALLBACK, + &&CASE_EEOP_PARAM_VARIABLE, &&CASE_EEOP_CASE_TESTVAL, &&CASE_EEOP_MAKE_READONLY, &&CASE_EEOP_IOCOERCE, @@ -1086,6 +1088,21 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull) EEO_NEXT(); } + EEO_CASE(EEOP_PARAM_VARIABLE) + { + /* + * direct access to session variable (without buffering). + * Because returned value can be used (without an assignement) + * after the referenced session variables is updated, we have + * to return copy of stored value every time. This is not an + * issue for local (plpgsql) variables. + */ + *op->resvalue = CopySessionVariableWithTypeCheck(op->d.vparam.varid, + op->resnull, + op->d.vparam.vartype); + EEO_NEXT(); + } + EEO_CASE(EEOP_CASE_TESTVAL) { /* diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index 0186be452c..a407b53038 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -48,6 +48,7 @@ #include "catalog/pg_publication.h" #include "commands/matview.h" #include "commands/trigger.h" +#include "commands/session_variable.h" #include "executor/execdebug.h" #include "executor/nodeSubplan.h" #include "foreign/fdwapi.h" @@ -201,6 +202,63 @@ standard_ExecutorStart(QueryDesc *queryDesc, int eflags) Assert(queryDesc->sourceText != NULL); estate->es_sourceText = queryDesc->sourceText; + /* + * The executor doesn't work with session variables directly. Values of + * related session variables are copied to dedicated array, and this array + * is passed to executor. + */ + if (queryDesc->num_session_variables > 0) + { + /* + * When a parallel query needs to access query parameters (including + * related session variables), then related session variables are + * restored (deserialized) in queryDesc already. So just push pointer + * of this array to executor's estate. + */ + Assert(IsParallelWorker()); + estate->es_session_variables = queryDesc->session_variables; + estate->es_num_session_variables = queryDesc->num_session_variables; + } + else if (queryDesc->plannedstmt->sessionVariables) + { + ListCell *lc; + int nSessionVariables; + int i = 0; + + /* + * In this case, the query uses session variables, but we have to + * prepare the array with passed values (of used session variables) + * first. + */ + Assert(!IsParallelWorker()); + nSessionVariables = list_length(queryDesc->plannedstmt->sessionVariables); + + /* Create the array used for passing values of used session variables */ + estate->es_session_variables = (SessionVariableValue *) + palloc(nSessionVariables * sizeof(SessionVariableValue)); + + /* Fill the array */ + foreach(lc, queryDesc->plannedstmt->sessionVariables) + { + AclResult aclresult; + Oid varid = lfirst_oid(lc); + + aclresult = object_aclcheck(VariableRelationId, varid, GetUserId(), ACL_SELECT); + if (aclresult != ACLCHECK_OK) + aclcheck_error(aclresult, OBJECT_VARIABLE, + get_session_variable_name(varid)); + + estate->es_session_variables[i].varid = varid; + estate->es_session_variables[i].value = CopySessionVariable(varid, + &estate->es_session_variables[i].isnull, + &estate->es_session_variables[i].typid); + + i++; + } + + estate->es_num_session_variables = nSessionVariables; + } + /* * Fill in the query environment, if any, from queryDesc. */ diff --git a/src/backend/executor/execParallel.c b/src/backend/executor/execParallel.c index cc2b8ccab7..64e3b66e73 100644 --- a/src/backend/executor/execParallel.c +++ b/src/backend/executor/execParallel.c @@ -12,8 +12,9 @@ * workers and ensuring that their state generally matches that of the * leader; see src/backend/access/transam/README.parallel for details. * However, we must save and restore relevant executor state, such as - * any ParamListInfo associated with the query, buffer/WAL usage info, and - * the actual plan to be passed down to the worker. + * any ParamListInfo associated with the query, buffer/WAL usage info, + * session variables buffer, and the actual plan to be passed down to + * the worker. * * IDENTIFICATION * src/backend/executor/execParallel.c @@ -66,6 +67,7 @@ #define PARALLEL_KEY_QUERY_TEXT UINT64CONST(0xE000000000000008) #define PARALLEL_KEY_JIT_INSTRUMENTATION UINT64CONST(0xE000000000000009) #define PARALLEL_KEY_WAL_USAGE UINT64CONST(0xE00000000000000A) +#define PARALLEL_KEY_SESSION_VARIABLES UINT64CONST(0xE00000000000000B) #define PARALLEL_TUPLE_QUEUE_SIZE 65536 @@ -140,6 +142,12 @@ static bool ExecParallelRetrieveInstrumentation(PlanState *planstate, /* Helper function that runs in the parallel worker. */ static DestReceiver *ExecParallelGetReceiver(dsm_segment *seg, shm_toc *toc); +/* Helper functions that can pass values of session variables */ +static Size EstimateSessionVariables(EState *estate); +static void SerializeSessionVariables(EState *estate, char **start_address); +static SessionVariableValue *RestoreSessionVariables(char **start_address, + int *num_session_variables); + /* * Create a serialized representation of the plan to be sent to each worker. */ @@ -598,6 +606,7 @@ ExecInitParallelPlan(PlanState *planstate, EState *estate, char *pstmt_data; char *pstmt_space; char *paramlistinfo_space; + char *session_variables_space; BufferUsage *bufusage_space; WalUsage *walusage_space; SharedExecutorInstrumentation *instrumentation = NULL; @@ -607,6 +616,7 @@ ExecInitParallelPlan(PlanState *planstate, EState *estate, int instrumentation_len = 0; int jit_instrumentation_len = 0; int instrument_offset = 0; + int session_variables_len = 0; Size dsa_minsize = dsa_minimum_size(); char *query_string; int query_len; @@ -662,6 +672,11 @@ ExecInitParallelPlan(PlanState *planstate, EState *estate, shm_toc_estimate_chunk(&pcxt->estimator, paramlistinfo_len); shm_toc_estimate_keys(&pcxt->estimator, 1); + /* Estimate space for serialized session variables. */ + session_variables_len = EstimateSessionVariables(estate); + shm_toc_estimate_chunk(&pcxt->estimator, session_variables_len); + shm_toc_estimate_keys(&pcxt->estimator, 1); + /* * Estimate space for BufferUsage. * @@ -756,6 +771,11 @@ ExecInitParallelPlan(PlanState *planstate, EState *estate, shm_toc_insert(pcxt->toc, PARALLEL_KEY_PARAMLISTINFO, paramlistinfo_space); SerializeParamList(estate->es_param_list_info, ¶mlistinfo_space); + /* Store serialized session variables. */ + session_variables_space = shm_toc_allocate(pcxt->toc, session_variables_len); + shm_toc_insert(pcxt->toc, PARALLEL_KEY_SESSION_VARIABLES, session_variables_space); + SerializeSessionVariables(estate, &session_variables_space); + /* Allocate space for each worker's BufferUsage; no need to initialize. */ bufusage_space = shm_toc_allocate(pcxt->toc, mul_size(sizeof(BufferUsage), pcxt->nworkers)); @@ -1403,6 +1423,7 @@ ParallelQueryMain(dsm_segment *seg, shm_toc *toc) SharedJitInstrumentation *jit_instrumentation; int instrument_options = 0; void *area_space; + char *sessionvariable_space; dsa_area *area; ParallelWorkerContext pwcxt; @@ -1428,6 +1449,14 @@ ParallelQueryMain(dsm_segment *seg, shm_toc *toc) area_space = shm_toc_lookup(toc, PARALLEL_KEY_DSA, false); area = dsa_attach_in_place(area_space, seg); + /* Reconstruct session variables. */ + sessionvariable_space = shm_toc_lookup(toc, + PARALLEL_KEY_SESSION_VARIABLES, + false); + queryDesc->session_variables = + RestoreSessionVariables(&sessionvariable_space, + &queryDesc->num_session_variables); + /* Start up the executor */ queryDesc->plannedstmt->jitFlags = fpes->jit_flags; ExecutorStart(queryDesc, fpes->eflags); @@ -1496,3 +1525,117 @@ ParallelQueryMain(dsm_segment *seg, shm_toc *toc) FreeQueryDesc(queryDesc); receiver->rDestroy(receiver); } + +/* + * Estimate the amount of space required to serialize a session variable. + */ +static Size +EstimateSessionVariables(EState *estate) +{ + int i; + Size sz = sizeof(int); + + if (estate->es_session_variables == NULL) + return sz; + + for (i = 0; i < estate->es_num_session_variables; i++) + { + SessionVariableValue *svarval; + Oid typeOid; + int16 typLen; + bool typByVal; + + svarval = &estate->es_session_variables[i]; + + typeOid = svarval->typid; + + sz = add_size(sz, sizeof(Oid)); /* space for type OID */ + + /* space for datum/isnull */ + Assert(OidIsValid(typeOid)); + get_typlenbyval(typeOid, &typLen, &typByVal); + + sz = add_size(sz, + datumEstimateSpace(svarval->value, svarval->isnull, typByVal, typLen)); + } + + return sz; +} + +/* + * Serialize a session variables buffer into caller-provided storage. + * + * We write the number of parameters first, as a 4-byte integer, and then + * write details for each parameter in turn. The details for each parameter + * consist of a 4-byte type OID, and then the datum as serialized by + * datumSerialize(). The caller is responsible for ensuring that there is + * enough storage to store the number of bytes that will be written; use + * EstimateSessionVariables to find out how many will be needed. + * *start_address is updated to point to the byte immediately following those + * written. + * + * RestoreSessionVariables can be used to recreate a session variable buffer + * based on the serialized representation; + */ +static void +SerializeSessionVariables(EState *estate, char **start_address) +{ + int nparams; + int i; + + /* Write number of parameters. */ + nparams = estate->es_num_session_variables; + memcpy(*start_address, &nparams, sizeof(int)); + *start_address += sizeof(int); + + /* Write each parameter in turn. */ + for (i = 0; i < nparams; i++) + { + SessionVariableValue *svarval; + Oid typeOid; + int16 typLen; + bool typByVal; + + svarval = &estate->es_session_variables[i]; + typeOid = svarval->typid; + + /* Write type OID. */ + memcpy(*start_address, &typeOid, sizeof(Oid)); + *start_address += sizeof(Oid); + + Assert(OidIsValid(typeOid)); + get_typlenbyval(typeOid, &typLen, &typByVal); + + datumSerialize(svarval->value, svarval->isnull, typByVal, typLen, + start_address); + } +} + +static SessionVariableValue * +RestoreSessionVariables(char **start_address, int *num_session_variables) +{ + SessionVariableValue *session_variables; + int i; + int nparams; + + memcpy(&nparams, *start_address, sizeof(int)); + *start_address += sizeof(int); + + *num_session_variables = nparams; + session_variables = (SessionVariableValue *) + palloc(nparams * sizeof(SessionVariableValue)); + + for (i = 0; i < nparams; i++) + { + SessionVariableValue *svarval = &session_variables[i]; + + /* Read type OID. */ + memcpy(&svarval->typid, *start_address, sizeof(Oid)); + *start_address += sizeof(Oid); + + /* Read datum/isnull. */ + svarval->value = datumRestore(start_address, &svarval->isnull); + } + + return session_variables; +} diff --git a/src/backend/jit/llvm/llvmjit_expr.c b/src/backend/jit/llvm/llvmjit_expr.c index 2c3d64ea6e..9f619a58c4 100644 --- a/src/backend/jit/llvm/llvmjit_expr.c +++ b/src/backend/jit/llvm/llvmjit_expr.c @@ -1073,6 +1073,12 @@ llvm_compile_expr(ExprState *state) LLVMBuildBr(b, opblocks[opno + 1]); break; + case EEOP_PARAM_VARIABLE: + build_EvalXFunc(b, mod, "ExecEvalParamVariable", + v_state, op, v_econtext); + LLVMBuildBr(b, opblocks[opno + 1]); + break; + case EEOP_PARAM_CALLBACK: { LLVMTypeRef v_functype; diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 1e4dd27dba..4157f755c9 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -321,6 +321,7 @@ standard_planner(Query *parse, const char *query_string, int cursorOptions, glob->lastPlanNodeId = 0; glob->transientPlan = false; glob->dependsOnRole = false; + glob->sessionVariables = NIL; /* * Assess whether it's feasible to use parallel mode for this query. We @@ -534,6 +535,7 @@ standard_planner(Query *parse, const char *query_string, int cursorOptions, result->paramExecTypes = glob->paramExecTypes; /* utilityStmt should be null, but we might as well copy it */ result->utilityStmt = parse->utilityStmt; + result->sessionVariables = glob->sessionVariables; result->stmt_location = parse->stmt_location; result->stmt_len = parse->stmt_len; @@ -700,6 +702,12 @@ subquery_planner(PlannerGlobal *glob, Query *parse, */ pull_up_subqueries(root); + /* + * Check if some subquery uses session variable. Flag hasSessionVariables + * should be true if query or some subquery uses any session variable. + */ + pull_up_has_session_variables(root); + /* * If this is a simple UNION ALL query, flatten it into an appendrel. We * do this now because it requires applying pull_up_subqueries to the leaf diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index f6f8a79354..f18ed8ef9a 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -211,6 +211,8 @@ static List *set_returning_clause_references(PlannerInfo *root, static List *set_windowagg_runcondition_references(PlannerInfo *root, List *runcondition, Plan *plan); +static bool pull_up_has_session_variables_walker(Node *node, + PlannerInfo *root); /***************************************************************************** @@ -1294,6 +1296,50 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) return plan; } +/* + * Search usage of session variables in subqueries + */ +void +pull_up_has_session_variables(PlannerInfo *root) +{ + Query *query = root->parse; + + if (query->hasSessionVariables) + { + root->hasSessionVariables = true; + } + else + { + (void) query_tree_walker(query, + pull_up_has_session_variables_walker, + (void *) root, 0); + } +} + +static bool +pull_up_has_session_variables_walker(Node *node, PlannerInfo *root) +{ + if (node == NULL) + return false; + if (IsA(node, Query)) + { + Query *query = (Query *) node; + + if (query->hasSessionVariables) + { + root->hasSessionVariables = true; + return false; + } + + /* Recurse into subselects */ + return query_tree_walker((Query *) node, + pull_up_has_session_variables_walker, + (void *) root, 0); + } + return expression_tree_walker(node, pull_up_has_session_variables_walker, + (void *) root); +} + /* * set_indexonlyscan_references * Do set_plan_references processing on an IndexOnlyScan @@ -1931,8 +1977,9 @@ copyVar(Var *var) * This is code that is common to all variants of expression-fixing. * We must look up operator opcode info for OpExpr and related nodes, * add OIDs from regclass Const nodes into root->glob->relationOids, and - * add PlanInvalItems for user-defined functions into root->glob->invalItems. - * We also fill in column index lists for GROUPING() expressions. + * add PlanInvalItems for user-defined functions and session variables into + * root->glob->invalItems. We also fill in column index lists for GROUPING() + * expressions. * * We assume it's okay to update opcode info in-place. So this could possibly * scribble on the planner's input data structures, but it's OK. @@ -2022,15 +2069,39 @@ fix_expr_common(PlannerInfo *root, Node *node) g->cols = cols; } } + else if (IsA(node, Param)) + { + Param *p = (Param *) node; + + if (p->paramkind == PARAM_VARIABLE) + { + PlanInvalItem *inval_item = makeNode(PlanInvalItem); + + /* paramid is still session variable id */ + inval_item->cacheId = VARIABLEOID; + inval_item->hashValue = GetSysCacheHashValue1(VARIABLEOID, + ObjectIdGetDatum(p->paramvarid)); + + /* Append this variable to global, register dependency */ + root->glob->invalItems = lappend(root->glob->invalItems, + inval_item); + } + } } /* * fix_param_node * Do set_plan_references processing on a Param + * Collect session variables list and replace variable oid by + * index to collected list. * * If it's a PARAM_MULTIEXPR, replace it with the appropriate Param from * root->multiexpr_params; otherwise no change is needed. * Just for paranoia's sake, we make a copy of the node in either case. + * + * If it's a PARAM_VARIABLE, then we collect used session variables in + * list root->glob->sessionVariable. We should to assign Param paramvarid + * too, and it is position of related session variable in mentioned list. */ static Node * fix_param_node(PlannerInfo *root, Param *p) @@ -2049,6 +2120,41 @@ fix_param_node(PlannerInfo *root, Param *p) elog(ERROR, "unexpected PARAM_MULTIEXPR ID: %d", p->paramid); return copyObject(list_nth(params, colno - 1)); } + + if (p->paramkind == PARAM_VARIABLE) + { + ListCell *lc; + int n = 0; + bool found = false; + + /* We will modify object */ + p = (Param *) copyObject(p); + + /* + * Now, we can actualize list of session variables, and we can + * complete paramid parameter. + */ + foreach(lc, root->glob->sessionVariables) + { + if (lfirst_oid(lc) == p->paramvarid) + { + p->paramid = n; + found = true; + break; + } + n += 1; + } + + if (!found) + { + root->glob->sessionVariables = lappend_oid(root->glob->sessionVariables, + p->paramvarid); + p->paramid = n; + } + + return (Node *) p; + } + return (Node *) copyObject(p); } @@ -2110,7 +2216,10 @@ fix_alternative_subplan(PlannerInfo *root, AlternativeSubPlan *asplan, * replacing Aggref nodes that should be replaced by initplan output Params, * choosing the best implementation for AlternativeSubPlans, * looking up operator opcode info for OpExpr and related nodes, - * and adding OIDs from regclass Const nodes into root->glob->relationOids. + * adding OIDs from regclass Const nodes into root->glob->relationOids, + * and assigning paramvarid to PARAM_VARIABLE params, and collecting + * of OIDs of session variables in root->glob->sessionVariables list + * (paramvarid is an position of related session variable in this list). * * 'node': the expression to be modified * 'rtoffset': how much to increment varnos by @@ -2132,7 +2241,8 @@ fix_scan_expr(PlannerInfo *root, Node *node, int rtoffset, double num_exec) root->multiexpr_params != NIL || root->glob->lastPHId != 0 || root->minmax_aggs != NIL || - root->hasAlternativeSubPlans) + root->hasAlternativeSubPlans || + root->hasSessionVariables) { return fix_scan_expr_mutator(node, &context); } diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c index 2f589b1b99..799033da11 100644 --- a/src/backend/optimizer/prep/prepjointree.c +++ b/src/backend/optimizer/prep/prepjointree.c @@ -1265,6 +1265,9 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, /* If subquery had any RLS conditions, now main query does too */ parse->hasRowSecurity |= subquery->hasRowSecurity; + /* If subquery had session variables, now main query does too */ + parse->hasSessionVariables |= subquery->hasSessionVariables; + /* * subquery won't be pulled up if it hasAggs, hasWindowFuncs, or * hasTargetSRFs, so no work needed on those flags diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c index 11269fee3e..fed4a6e2f4 100644 --- a/src/backend/optimizer/util/clauses.c +++ b/src/backend/optimizer/util/clauses.c @@ -26,6 +26,7 @@ #include "catalog/pg_operator.h" #include "catalog/pg_proc.h" #include "catalog/pg_type.h" +#include "commands/session_variable.h" #include "executor/executor.h" #include "executor/functions.h" #include "funcapi.h" @@ -833,16 +834,17 @@ max_parallel_hazard_walker(Node *node, max_parallel_hazard_context *context) /* * We can't pass Params to workers at the moment either, so they are also - * parallel-restricted, unless they are PARAM_EXTERN Params or are - * PARAM_EXEC Params listed in safe_param_ids, meaning they could be - * either generated within workers or can be computed by the leader and - * then their value can be passed to workers. + * parallel-restricted, unless they are PARAM_EXTERN or PARAM_VARIABLE + * Params or are PARAM_EXEC Params listed in safe_param_ids, meaning they + * could be either generated within workers or can be computed by the + * leader and then their value can be passed to workers. */ else if (IsA(node, Param)) { Param *param = (Param *) node; - if (param->paramkind == PARAM_EXTERN) + if (param->paramkind == PARAM_EXTERN || + param->paramkind == PARAM_VARIABLE) return false; if (param->paramkind != PARAM_EXEC || @@ -2298,6 +2300,7 @@ convert_saop_to_hashed_saop_walker(Node *node, void *context) * value of the Param. * 2. Fold stable, as well as immutable, functions to constants. * 3. Reduce PlaceHolderVar nodes to their contained expressions. + * 4. Current value of session variable can be used for estimation too. *-------------------- */ Node * @@ -2420,6 +2423,29 @@ eval_const_expressions_mutator(Node *node, } } } + else if (param->paramkind == PARAM_VARIABLE && + context->estimate) + { + int16 typLen; + bool typByVal; + Datum pval; + bool isnull; + + get_typlenbyval(param->paramtype, + &typLen, &typByVal); + + pval = CopySessionVariableWithTypeCheck(param->paramvarid, + &isnull, + param->paramtype); + + return (Node *) makeConst(param->paramtype, + param->paramtypmod, + param->paramcollid, + (int) typLen, + pval, + isnull, + typByVal); + } /* * Not replaceable, so just copy the Param (no need to @@ -4803,21 +4829,43 @@ substitute_actual_parameters_mutator(Node *node, { if (node == NULL) return NULL; + + /* + * SQL functions can contain two different kind of params. The nodes with + * paramkind PARAM_EXTERN are related to function's arguments (and should + * be replaced in this step), because this is how we apply the function's + * arguments for an expression. + * + * The nodes with paramkind PARAM_VARIABLE are related to usage of session + * variables. The values of session variables are not passed to expression + * by expression arguments, so it should not be replaced here by + * function's arguments. Although we could substitute params related to + * immutable session variables with default expression by this default + * expression, it is safer to not do it. This way we don't have to run + * security checks here. There can be some performance loss, but an access + * to session variable is fast (and the result of default expression is + * immediately materialized and can be reused). + */ if (IsA(node, Param)) { Param *param = (Param *) node; - if (param->paramkind != PARAM_EXTERN) + if (param->paramkind != PARAM_EXTERN && + param->paramkind != PARAM_VARIABLE) elog(ERROR, "unexpected paramkind: %d", (int) param->paramkind); - if (param->paramid <= 0 || param->paramid > context->nargs) - elog(ERROR, "invalid paramid: %d", param->paramid); - /* Count usage of parameter */ - context->usecounts[param->paramid - 1]++; + if (param->paramkind == PARAM_EXTERN) + { + if (param->paramid <= 0 || param->paramid > context->nargs) + elog(ERROR, "invalid paramid: %d", param->paramid); + + /* Count usage of parameter */ + context->usecounts[param->paramid - 1]++; - /* Select the appropriate actual arg and replace the Param with it */ - /* We don't need to copy at this time (it'll get done later) */ - return list_nth(context->args, param->paramid - 1); + /* Select the appropriate actual arg and replace the Param with it */ + /* We don't need to copy at this time (it'll get done later) */ + return list_nth(context->args, param->paramid - 1); + } } return expression_tree_mutator(node, substitute_actual_parameters_mutator, (void *) context); diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 70932dba61..c2f4f4836f 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -527,6 +527,8 @@ transformDeleteStmt(ParseState *pstate, DeleteStmt *stmt) qry->hasTargetSRFs = pstate->p_hasTargetSRFs; qry->hasAggs = pstate->p_hasAggs; + qry->hasSessionVariables = pstate->p_hasSessionVariables; + assign_query_collations(pstate, qry); /* this must be done after collations, for reliable comparison of exprs */ @@ -951,6 +953,7 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) qry->hasTargetSRFs = pstate->p_hasTargetSRFs; qry->hasSubLinks = pstate->p_hasSubLinks; + qry->hasSessionVariables = pstate->p_hasSessionVariables; assign_query_collations(pstate, qry); @@ -1405,6 +1408,7 @@ transformSelectStmt(ParseState *pstate, SelectStmt *stmt) qry->hasWindowFuncs = pstate->p_hasWindowFuncs; qry->hasTargetSRFs = pstate->p_hasTargetSRFs; qry->hasAggs = pstate->p_hasAggs; + qry->hasSessionVariables = pstate->p_hasSessionVariables; foreach(l, stmt->lockingClause) { @@ -1631,6 +1635,7 @@ transformValuesClause(ParseState *pstate, SelectStmt *stmt) qry->jointree = makeFromExpr(pstate->p_joinlist, NULL); qry->hasSubLinks = pstate->p_hasSubLinks; + qry->hasSessionVariables = pstate->p_hasSessionVariables; assign_query_collations(pstate, qry); @@ -1881,6 +1886,7 @@ transformSetOperationStmt(ParseState *pstate, SelectStmt *stmt) qry->hasWindowFuncs = pstate->p_hasWindowFuncs; qry->hasTargetSRFs = pstate->p_hasTargetSRFs; qry->hasAggs = pstate->p_hasAggs; + qry->hasSessionVariables = pstate->p_hasSessionVariables; foreach(l, lockingClause) { @@ -2421,6 +2427,7 @@ transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt) qry->hasTargetSRFs = pstate->p_hasTargetSRFs; qry->hasSubLinks = pstate->p_hasSubLinks; + qry->hasSessionVariables = pstate->p_hasSessionVariables; assign_query_collations(pstate, qry); diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index c69259c9e3..f40f62425e 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -35,16 +35,18 @@ #include "parser/parse_relation.h" #include "parser/parse_target.h" #include "parser/parse_type.h" +#include "storage/lmgr.h" #include "utils/builtins.h" #include "utils/date.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" #include "utils/timestamp.h" +#include "utils/typcache.h" #include "utils/xml.h" /* GUC parameters */ bool Transform_null_equals = false; - +bool session_variables_ambiguity_warning = false; static Node *transformExprRecurse(ParseState *pstate, Node *expr); static Node *transformParamRef(ParseState *pstate, ParamRef *pref); @@ -93,6 +95,9 @@ static Expr *make_distinct_op(ParseState *pstate, List *opname, Node *ltree, Node *rtree, int location); static Node *make_nulltest_from_distinct(ParseState *pstate, A_Expr *distincta, Node *arg); +static Node *makeParamSessionVariable(ParseState *pstate, + Oid varid, Oid typid, int32 typmod, Oid collid, + char *attrname, int location); /* @@ -460,6 +465,88 @@ transformIndirection(ParseState *pstate, A_Indirection *ind) return result; } +/* + * Returns true if the given expression kind is valid for session variables + * Session variables can be used everywhere where external parameters can be + * used. Session variables are not allowed in DDL commands or in constraints. + * + * An identifier can be parsed as a session variable only for expression kinds + * where session variables are allowed. This is the primary usage of this + * function. + * + * Second usage of this function is to decide whether "column does not exist" or + * "column or variable does not exist" error message should be printed. + * When we are in an expression where session variables cannot be used, we raise + * the first form of error message. + */ +static bool +expr_kind_allows_session_variables(ParseExprKind p_expr_kind) +{ + bool result = false; + + switch (p_expr_kind) + { + case EXPR_KIND_NONE: + Assert(false); /* can't happen */ + return false; + + /* allow */ + case EXPR_KIND_OTHER: + case EXPR_KIND_JOIN_ON: + case EXPR_KIND_FROM_SUBSELECT: + case EXPR_KIND_FROM_FUNCTION: + case EXPR_KIND_WHERE: + case EXPR_KIND_HAVING: + case EXPR_KIND_FILTER: + case EXPR_KIND_WINDOW_PARTITION: + case EXPR_KIND_WINDOW_ORDER: + case EXPR_KIND_WINDOW_FRAME_RANGE: + case EXPR_KIND_WINDOW_FRAME_ROWS: + case EXPR_KIND_WINDOW_FRAME_GROUPS: + case EXPR_KIND_SELECT_TARGET: + case EXPR_KIND_INSERT_TARGET: + case EXPR_KIND_UPDATE_SOURCE: + case EXPR_KIND_UPDATE_TARGET: + case EXPR_KIND_MERGE_WHEN: + case EXPR_KIND_GROUP_BY: + case EXPR_KIND_ORDER_BY: + case EXPR_KIND_DISTINCT_ON: + case EXPR_KIND_LIMIT: + case EXPR_KIND_OFFSET: + case EXPR_KIND_RETURNING: + case EXPR_KIND_VALUES: + case EXPR_KIND_VALUES_SINGLE: + case EXPR_KIND_ALTER_COL_TRANSFORM: + case EXPR_KIND_EXECUTE_PARAMETER: + case EXPR_KIND_POLICY: + case EXPR_KIND_CALL_ARGUMENT: + case EXPR_KIND_COPY_WHERE: + case EXPR_KIND_LET_TARGET: + result = true; + break; + + /* not allow */ + case EXPR_KIND_CHECK_CONSTRAINT: + case EXPR_KIND_DOMAIN_CHECK: + case EXPR_KIND_COLUMN_DEFAULT: + case EXPR_KIND_FUNCTION_DEFAULT: + case EXPR_KIND_INDEX_EXPRESSION: + case EXPR_KIND_INDEX_PREDICATE: + case EXPR_KIND_STATS_EXPRESSION: + case EXPR_KIND_TRIGGER_WHEN: + case EXPR_KIND_PARTITION_BOUND: + case EXPR_KIND_PARTITION_EXPRESSION: + case EXPR_KIND_GENERATED_COLUMN: + case EXPR_KIND_VARIABLE_DEFAULT: + case EXPR_KIND_JOIN_USING: + case EXPR_KIND_CYCLE_MARK: + result = false; + break; + } + + return result; +} + /* * Transform a ColumnRef. * @@ -808,6 +895,104 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) parser_errposition(pstate, cref->location))); } + /* + * There are contexts where session's variables are not allowed. The + * question is if we want to identify session's variables in these + * contexts? The code can be more simple, when we don't do it, but then we + * cannot to raise maybe useful message like "you cannot to use session + * variables here". On second hand, in this case the warnings about + * session's variable shadowing can be messy. + */ + if (expr_kind_allows_session_variables(pstate->p_expr_kind)) + { + Oid varid = InvalidOid; + char *attrname = NULL; + bool not_unique; + + /* + * Session variables are shadowed by columns, routine's variables or + * routine's arguments ever. We don't want to use session variable + * when it is not exactly shadowed, but RTE is valid like: + * + * CREATE TYPE T AS (c int); CREATE VARIABLE foo AS T; CREATE TABLE + * foo(a int, b int); + * + * SELECT foo.a, foo.b, foo.c FROM foo; + * + * This case can be messy and then we disallow it. When we know, so + * possible variable will be shadowed, we try to identify variable + * only when session_variables_ambiguity_warning is requested. + */ + if (node || + (!node && relname && crerr == CRERR_NO_COLUMN)) + { + /* + * In this path we just try (if it is wanted) detect if session + * variable is shadowed. + */ + if (session_variables_ambiguity_warning) + { + /* + * The AccessShareLock is created on related session variable. The lock + * will be kept for the whole transaction. + */ + varid = IdentifyVariable(cref->fields, &attrname, ¬_unique, true); + + if (OidIsValid(varid)) + { + /* This path will ending by WARNING. Unlock variable first */ + UnlockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); + + if (node) + ereport(WARNING, + (errcode(ERRCODE_AMBIGUOUS_COLUMN), + errmsg("session variable \"%s\" is shadowed", + NameListToString(cref->fields)), + errdetail("Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name."), + parser_errposition(pstate, cref->location))); + else + /* session variable is shadowed by RTE */ + ereport(WARNING, + (errcode(ERRCODE_AMBIGUOUS_COLUMN), + errmsg("session variable \"%s.%s\" is shadowed", + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)), + errdetail("Session variables can be shadowed by tables or table's aliases with the same name."), + parser_errposition(pstate, cref->location))); + } + } + } + else + { + /* + * The AccessShareLock is created on related session variable. The lock + * will be kept for the whole transaction. + */ + varid = IdentifyVariable(cref->fields, &attrname, ¬_unique, false); + + if (OidIsValid(varid)) + { + Oid typid; + int32 typmod; + Oid collid; + + if (not_unique) + ereport(ERROR, + (errcode(ERRCODE_AMBIGUOUS_PARAMETER), + errmsg("session variable reference \"%s\" is ambiguous", + NameListToString(cref->fields)), + parser_errposition(pstate, cref->location))); + + get_session_variable_type_typmod_collid(varid, &typid, &typmod, + &collid); + + node = makeParamSessionVariable(pstate, + varid, typid, typmod, collid, + attrname, cref->location); + } + } + } + /* * Throw error if no translation found. */ @@ -842,6 +1027,72 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) return node; } +/* + * Generate param variable for reference to session variable + */ +static Node * +makeParamSessionVariable(ParseState *pstate, + Oid varid, Oid typid, int32 typmod, Oid collid, + char *attrname, int location) +{ + Param *param; + + param = makeNode(Param); + + param->paramkind = PARAM_VARIABLE; + param->paramvarid = varid; + param->paramtype = typid; + param->paramtypmod = typmod; + param->paramcollid = collid; + + pstate->p_hasSessionVariables = true; + + if (attrname != NULL) + { + TupleDesc tupdesc; + int i; + + tupdesc = lookup_rowtype_tupdesc_noerror(typid, typmod, true); + if (!tupdesc) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("variable \"%s.%s\" is of type \"%s\", which is not a composite type", + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid), + format_type_be(typid)), + parser_errposition(pstate, location))); + + for (i = 0; i < tupdesc->natts; i++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, i); + + if (strcmp(attrname, NameStr(att->attname)) == 0 && + !att->attisdropped) + { + /* Success, so generate a FieldSelect expression */ + FieldSelect *fselect = makeNode(FieldSelect); + + fselect->arg = (Expr *) param; + fselect->fieldnum = i + 1; + fselect->resulttype = att->atttypid; + fselect->resulttypmod = att->atttypmod; + /* save attribute's collation for parse_collate.c */ + fselect->resultcollid = att->attcollation; + + ReleaseTupleDesc(tupdesc); + return (Node *) fselect; + } + } + + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_COLUMN), + errmsg("could not identify column \"%s\" in variable", attrname), + parser_errposition(pstate, location))); + } + + return (Node *) param; +} + static Node * transformParamRef(ParseState *pstate, ParamRef *pref) { diff --git a/src/backend/tcop/pquery.c b/src/backend/tcop/pquery.c index 5565f200c3..5382c450dc 100644 --- a/src/backend/tcop/pquery.c +++ b/src/backend/tcop/pquery.c @@ -86,6 +86,9 @@ CreateQueryDesc(PlannedStmt *plannedstmt, qd->queryEnv = queryEnv; qd->instrument_options = instrument_options; /* instrumentation wanted? */ + qd->num_session_variables = 0; + qd->session_variables = NULL; + /* null these fields until set by ExecutorStart */ qd->tupDesc = NULL; qd->estate = NULL; diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c index 60f9d08d5d..8b37318254 100644 --- a/src/backend/utils/adt/ruleutils.c +++ b/src/backend/utils/adt/ruleutils.c @@ -38,6 +38,7 @@ #include "catalog/pg_statistic_ext.h" #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "commands/defrem.h" #include "commands/tablespace.h" #include "common/keywords.h" @@ -515,6 +516,7 @@ static char *generate_function_name(Oid funcid, int nargs, static char *generate_operator_name(Oid operid, Oid arg1, Oid arg2); static void add_cast_to(StringInfo buf, Oid typid); static char *generate_qualified_type_name(Oid typid); +static char *generate_session_variable_name(Oid varid); static text *string_to_text(char *str); static char *flatten_reloptions(Oid relid); static void get_reloptions(StringInfo buf, Datum reloptions); @@ -8148,6 +8150,14 @@ get_parameter(Param *param, deparse_context *context) return; } + /* translate paramvarid to session variable name */ + if (param->paramkind == PARAM_VARIABLE) + { + appendStringInfo(context->buf, "%s", + generate_session_variable_name(param->paramvarid)); + return; + } + /* * If it's an external parameter, see if the outermost namespace provides * function argument names. @@ -12451,6 +12461,42 @@ generate_collation_name(Oid collid) return result; } +/* + * generate_session_variable_name + * Compute the name to display for a session variable specified by OID + * + * The result includes all necessary quoting and schema-prefixing. + */ +static char * +generate_session_variable_name(Oid varid) +{ + HeapTuple tup; + Form_pg_variable varform; + char *varname; + char *nspname; + char *result; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for variable %u", varid); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + varname = NameStr(varform->varname); + + if (!VariableIsVisible(varid)) + nspname = get_namespace_name_or_temp(varform->varnamespace); + else + nspname = NULL; + + result = quote_qualified_identifier(nspname, varname); + + ReleaseSysCache(tup); + + return result; +} + /* * Given a C string, produce a TEXT datum. * diff --git a/src/backend/utils/cache/plancache.c b/src/backend/utils/cache/plancache.c index 87210fcf62..a3bea4e951 100644 --- a/src/backend/utils/cache/plancache.c +++ b/src/backend/utils/cache/plancache.c @@ -58,6 +58,7 @@ #include "access/transam.h" #include "catalog/namespace.h" +#include "catalog/pg_variable.h" #include "executor/executor.h" #include "miscadmin.h" #include "nodes/nodeFuncs.h" @@ -1874,9 +1875,12 @@ ScanQueryForLocks(Query *parsetree, bool acquire) /* * Recurse into sublink subqueries, too. But we already did the ones in - * the rtable and cteList. + * the rtable and cteList. We need to force recursive call for session + * variables too, to find and lock variables used in query (see + * ScanQueryWalker). */ - if (parsetree->hasSubLinks) + if (parsetree->hasSubLinks || + parsetree->hasSessionVariables) { query_tree_walker(parsetree, ScanQueryWalker, (void *) &acquire, @@ -1885,7 +1889,8 @@ ScanQueryForLocks(Query *parsetree, bool acquire) } /* - * Walker to find sublink subqueries for ScanQueryForLocks + * Walker to find sublink subqueries or referenced session variables + * for ScanQueryForLocks */ static bool ScanQueryWalker(Node *node, bool *acquire) @@ -1900,6 +1905,20 @@ ScanQueryWalker(Node *node, bool *acquire) ScanQueryForLocks(castNode(Query, sub->subselect), *acquire); /* Fall through to process lefthand args of SubLink */ } + else if (IsA(node, Param)) + { + Param *p = (Param *) node; + + if (p->paramkind == PARAM_VARIABLE) + { + if (acquire) + LockDatabaseObject(VariableRelationId, p->paramvarid, + 0, AccessShareLock); + else + UnlockDatabaseObject(VariableRelationId, p->paramvarid, + 0, AccessShareLock); + } + } /* * Do NOT recurse into Query nodes, because ScanQueryForLocks already @@ -2031,7 +2050,9 @@ PlanCacheRelCallback(Datum arg, Oid relid) /* * PlanCacheObjectCallback - * Syscache inval callback function for PROCOID and TYPEOID caches + * Syscache inval callback function for TYPEOID, PROCOID, NAMESPACEOID, + * OPEROID, AMOPOPID, FOREIGNSERVEROID, FOREIGNDATAWRAPPEROID and VARIABLEOID + * caches. * * Invalidate all plans mentioning the object with the specified hash value, * or all plans mentioning any member of this cache if hashvalue == 0. diff --git a/src/backend/utils/fmgr/fmgr.c b/src/backend/utils/fmgr/fmgr.c index f72dd25efa..006bfc76d7 100644 --- a/src/backend/utils/fmgr/fmgr.c +++ b/src/backend/utils/fmgr/fmgr.c @@ -1998,9 +1998,13 @@ get_call_expr_arg_stable(Node *expr, int argnum) */ if (IsA(arg, Const)) return true; - if (IsA(arg, Param) && - ((Param *) arg)->paramkind == PARAM_EXTERN) - return true; + if (IsA(arg, Param)) + { + Param *p = (Param *) arg; + + if (p->paramkind == PARAM_EXTERN || p->paramkind == PARAM_VARIABLE) + return true; + } return false; } diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 8abf9bb644..37b8373e04 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -1539,6 +1539,16 @@ struct config_bool ConfigureNamesBool[] = false, NULL, NULL, NULL }, + { + {"session_variables_ambiguity_warning", PGC_USERSET, DEVELOPER_OPTIONS, + gettext_noop("Raise warning when reference to a session variable is ambiguous."), + NULL, + GUC_NOT_IN_SAMPLE + }, + &session_variables_ambiguity_warning, + false, + NULL, NULL, NULL + }, { {"db_user_namespace", PGC_SIGHUP, CONN_AUTH_AUTH, gettext_noop("Enables per-database user names."), diff --git a/src/include/catalog/namespace.h b/src/include/catalog/namespace.h index 787de15ed1..312ecfe5b3 100644 --- a/src/include/catalog/namespace.h +++ b/src/include/catalog/namespace.h @@ -168,6 +168,7 @@ extern void ResetTempTableNamespace(void); extern List *NamesFromList(List *names); extern Oid LookupVariable(const char *nspname, const char *varname, bool missing_ok); +extern Oid IdentifyVariable(List *names, char **attrname, bool *not_unique, bool noerror); extern OverrideSearchPath *GetOverrideSearchPath(MemoryContext context); extern OverrideSearchPath *CopyOverrideSearchPath(OverrideSearchPath *path); diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 258a2ed903..e03f4b3eb9 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12061,7 +12061,6 @@ proname => 'brin_minmax_multi_summary_send', provolatile => 's', prorettype => 'bytea', proargtypes => 'pg_brin_minmax_multi_summary', prosrc => 'brin_minmax_multi_summary_send' }, - { oid => '8981', descr => 'arbitrary value from among input values', proname => 'any_value', prokind => 'a', proisstrict => 'f', prorettype => 'anyelement', proargtypes => 'anyelement', @@ -12069,5 +12068,11 @@ { oid => '8982', descr => 'aggregate transition function', proname => 'any_value_transfn', prorettype => 'anyelement', proargtypes => 'anyelement anyelement', prosrc => 'any_value_transfn' }, - +{ oid => '8488', descr => 'list of used session variables', + proname => 'pg_session_variables', prorows => '1000', proretset => 't', + provolatile => 's', prorettype => 'record', proargtypes => '', + proallargtypes => '{oid,text,text,oid,text,bool,bool,bool,bool}', + proargmodes => '{o,o,o,o,o,o,o,o,o}', + proargnames => '{varid,schema,name,typid,typname,removed,has_value,can_select,can_update}', + prosrc => 'pg_session_variables' }, ] diff --git a/src/include/commands/session_variable.h b/src/include/commands/session_variable.h index 343ee070a5..f442c5d3b8 100644 --- a/src/include/commands/session_variable.h +++ b/src/include/commands/session_variable.h @@ -1,7 +1,7 @@ /*------------------------------------------------------------------------- * - * sessionvariable.h - * prototypes for sessionvariable.c. + * session_variable.h + * prototypes for session_variable.c. * * * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group @@ -26,6 +26,14 @@ extern void SessionVariableCreatePostprocess(Oid varid, char eoxaction); extern void SessionVariableDropPostprocess(Oid varid); +extern Datum CopySessionVariable(Oid varid, bool *isNull, Oid *typid); +extern Datum CopySessionVariableWithTypeCheck(Oid varid, bool *isNull, Oid expected_typid); + +extern void SetSessionVariable(Oid varid, Datum value, bool isNull); +extern void SetSessionVariableWithSecurityCheck(Oid varid, Datum value, bool isNull); + +extern void ResetSessionVariables(void); + extern void AtPreEOXact_SessionVariable(bool isCommit); extern void AtEOSubXact_SessionVariable(bool isCommit, SubTransactionId mySubid, diff --git a/src/include/executor/execExpr.h b/src/include/executor/execExpr.h index 157b0d85f2..b3ddf2f8db 100644 --- a/src/include/executor/execExpr.h +++ b/src/include/executor/execExpr.h @@ -159,6 +159,7 @@ typedef enum ExprEvalOp EEOP_PARAM_EXEC, EEOP_PARAM_EXTERN, EEOP_PARAM_CALLBACK, + EEOP_PARAM_VARIABLE, /* return CaseTestExpr value */ EEOP_CASE_TESTVAL, @@ -384,6 +385,13 @@ typedef struct ExprEvalStep Oid paramtype; /* OID of parameter's datatype */ } param; + /* for EEOP_PARAM_VARIABLE */ + struct + { + Oid varid; /* OID of assigned variable */ + Oid vartype; /* OID of parameter's datatype */ + } vparam; + /* for EEOP_PARAM_CALLBACK */ struct { @@ -769,6 +777,9 @@ extern void ExecEvalParamExec(ExprState *state, ExprEvalStep *op, ExprContext *econtext); extern void ExecEvalParamExtern(ExprState *state, ExprEvalStep *op, ExprContext *econtext); +extern void ExecEvalParamVariable(ExprState *state, ExprEvalStep *op, + ExprContext *econtext); +extern void ExecEvalSQLValueFunction(ExprState *state, ExprEvalStep *op); extern void ExecEvalCurrentOfExpr(ExprState *state, ExprEvalStep *op); extern void ExecEvalNextValueExpr(ExprState *state, ExprEvalStep *op); extern void ExecEvalRowNull(ExprState *state, ExprEvalStep *op, diff --git a/src/include/executor/execdesc.h b/src/include/executor/execdesc.h index af2bf36dfb..c4c6331774 100644 --- a/src/include/executor/execdesc.h +++ b/src/include/executor/execdesc.h @@ -48,6 +48,10 @@ typedef struct QueryDesc EState *estate; /* executor's query-wide state */ PlanState *planstate; /* tree of per-plan-node state */ + /* reference to session variables buffer */ + int num_session_variables; + SessionVariableValue *session_variables; + /* This field is set by ExecutorRun */ bool already_executed; /* true if previously executed */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 61b3517906..5a464c12e5 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -601,6 +601,18 @@ typedef struct AsyncRequest * tuples) */ } AsyncRequest; +/* ---------------- + * SessionVariableValue + * ---------------- + */ +typedef struct SessionVariableValue +{ + Oid varid; + Oid typid; + bool isnull; + Datum value; +} SessionVariableValue; + /* ---------------- * EState information * @@ -653,6 +665,13 @@ typedef struct EState ParamListInfo es_param_list_info; /* values of external params */ ParamExecData *es_param_exec_vals; /* values of internal params */ + /* Variables info: */ + /* number of used session variables */ + int es_num_session_variables; + + /* array of copied values of session variables */ + SessionVariableValue *es_session_variables; + QueryEnvironment *es_queryEnv; /* query environment */ /* Other working state: */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index eecbbd7768..1f067fc84f 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -167,6 +167,8 @@ typedef struct Query bool hasForUpdate pg_node_attr(query_jumble_ignore); /* rewriter has applied some RLS policy */ bool hasRowSecurity pg_node_attr(query_jumble_ignore); + /* uses session variables */ + bool hasSessionVariables pg_node_attr(query_jumble_ignore); /* is a RETURN statement */ bool isReturn pg_node_attr(query_jumble_ignore); diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 7d4f24d250..4f698556a2 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -160,6 +160,9 @@ typedef struct PlannerGlobal /* partition descriptors */ PartitionDirectory partition_directory pg_node_attr(read_write_ignore); + + /* list of used session variables */ + List *sessionVariables; } PlannerGlobal; /* macro for fetching the Plan associated with a SubPlan node */ @@ -499,6 +502,8 @@ struct PlannerInfo bool placeholdersFrozen; /* true if planning a recursive WITH item */ bool hasRecursion; + /* true if session variables were used */ + bool hasSessionVariables; /* * Information about aggregates. Filled by preprocess_aggrefs(). diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index 1b787fe031..d287a80dce 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -95,6 +95,8 @@ typedef struct PlannedStmt Node *utilityStmt; /* non-null if this is utility stmt */ + List *sessionVariables; /* OIDs for PARAM_VARIABLE Params */ + /* statement location in source string (copied from Query) */ int stmt_location; /* start location, or -1 if unknown */ int stmt_len; /* length in bytes; 0 means "rest of string" */ diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h index be9c29f0bf..3dd80d8534 100644 --- a/src/include/nodes/primnodes.h +++ b/src/include/nodes/primnodes.h @@ -43,7 +43,9 @@ typedef struct Alias List *colnames; /* optional list of column aliases */ } Alias; -/* What to do at commit time for temporary relations */ +/* + * What to do at commit time for temporary relations or session variables. + */ typedef enum OnCommitAction { ONCOMMIT_NOOP, /* No ON COMMIT clause (do nothing) */ @@ -339,13 +341,17 @@ typedef struct Const * of the `paramid' field contain the SubLink's subLinkId, and * the low-order 16 bits contain the column number. (This type * of Param is also converted to PARAM_EXEC during planning.) + * + * PARAM_VARIABLE: The parameter is an access to session variable + * paramid holds varid. */ typedef enum ParamKind { PARAM_EXTERN, PARAM_EXEC, PARAM_SUBLINK, - PARAM_MULTIEXPR + PARAM_MULTIEXPR, + PARAM_VARIABLE } ParamKind; typedef struct Param @@ -358,6 +364,8 @@ typedef struct Param int32 paramtypmod pg_node_attr(query_jumble_ignore); /* OID of collation, or InvalidOid if none */ Oid paramcollid pg_node_attr(query_jumble_ignore); + /* OID of session variable if it is used */ + Oid paramvarid; /* token location, or -1 if unknown */ int location; } Param; diff --git a/src/include/optimizer/planmain.h b/src/include/optimizer/planmain.h index 5fc900737d..f106768beb 100644 --- a/src/include/optimizer/planmain.h +++ b/src/include/optimizer/planmain.h @@ -114,4 +114,6 @@ extern void record_plan_function_dependency(PlannerInfo *root, Oid funcid); extern void record_plan_type_dependency(PlannerInfo *root, Oid typid); extern bool extract_query_dependencies_walker(Node *node, PlannerInfo *context); +extern void pull_up_has_session_variables(PlannerInfo *root); + #endif /* PLANMAIN_H */ diff --git a/src/include/parser/parse_expr.h b/src/include/parser/parse_expr.h index 7d38ca75f7..f6a5ccb0c1 100644 --- a/src/include/parser/parse_expr.h +++ b/src/include/parser/parse_expr.h @@ -17,6 +17,7 @@ /* GUC parameters */ extern PGDLLIMPORT bool Transform_null_equals; +extern PGDLLIMPORT bool session_variables_ambiguity_warning; extern Node *transformExpr(ParseState *pstate, Node *expr, ParseExprKind exprKind); diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h index eccde83628..4e9692d05a 100644 --- a/src/include/parser/parse_node.h +++ b/src/include/parser/parse_node.h @@ -225,6 +225,7 @@ struct ParseState bool p_hasTargetSRFs; bool p_hasSubLinks; bool p_hasModifyingCTE; + bool p_hasSessionVariables; Node *p_last_srf; /* most recent set-returning func/op found */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 88aa4a29c2..ad3eec1f74 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2481,6 +2481,7 @@ SerializedTransactionState Session SessionBackupState SessionEndType +SessionVariableValue SetConstraintState SetConstraintStateData SetConstraintTriggerData @@ -2665,6 +2666,8 @@ SupportRequestRows SupportRequestSelectivity SupportRequestSimplify SupportRequestWFuncMonotonic +SVariable +SVariableData SVariableXActAction SVariableXActActionItem Syn -- 2.40.1 [text/x-patch] v20230516-1-0001-catalog-support-for-session-variables.patch (85.2K, ../../CAFj8pRDF_eygHWcG8Y9d73zmus=ohVup-ngP7w-m4tbtWg14KQ@mail.gmail.com/12-v20230516-1-0001-catalog-support-for-session-variables.patch) download | inline diff: From 290fef07d9d233c328aa308fea217f604638bd6d Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Sun, 13 Nov 2022 17:39:37 +0100 Subject: [PATCH 01/10] catalog support for session variables Implementation new system object class - session variable with new access rights SELECT, UPDATE, with routines for creating session variable, initialization of session variable from system catalog, and lookups routines for identification of session variables. --- src/backend/access/transam/xact.c | 11 + src/backend/catalog/Makefile | 4 +- src/backend/catalog/aclchk.c | 77 +++++ src/backend/catalog/dependency.c | 13 +- src/backend/catalog/meson.build | 1 + src/backend/catalog/namespace.c | 140 +++++++++ src/backend/catalog/objectaddress.c | 122 +++++++- src/backend/catalog/pg_shdepend.c | 2 + src/backend/catalog/pg_variable.c | 378 ++++++++++++++++++++++++ src/backend/commands/Makefile | 1 + src/backend/commands/alter.c | 9 + src/backend/commands/dropcmds.c | 4 + src/backend/commands/event_trigger.c | 6 + src/backend/commands/seclabel.c | 1 + src/backend/commands/session_variable.c | 243 +++++++++++++++ src/backend/commands/tablecmds.c | 43 +++ src/backend/parser/gram.y | 145 ++++++++- src/backend/parser/parse_agg.c | 2 + src/backend/parser/parse_expr.c | 5 + src/backend/parser/parse_func.c | 1 + src/backend/parser/parse_utilcmd.c | 12 + src/backend/tcop/utility.c | 16 + src/backend/utils/adt/acl.c | 7 + src/backend/utils/cache/lsyscache.c | 113 +++++++ src/backend/utils/cache/syscache.c | 14 + src/include/catalog/dependency.h | 5 +- src/include/catalog/meson.build | 1 + src/include/catalog/namespace.h | 5 + src/include/catalog/pg_default_acl.h | 1 + src/include/catalog/pg_proc.dat | 3 + src/include/catalog/pg_variable.h | 130 ++++++++ src/include/commands/session_variable.h | 34 +++ src/include/nodes/parsenodes.h | 20 ++ src/include/parser/kwlist.h | 2 + src/include/parser/parse_node.h | 1 + src/include/tcop/cmdtaglist.h | 3 + src/include/utils/acl.h | 1 + src/include/utils/lsyscache.h | 9 + src/include/utils/syscache.h | 6 +- src/test/regress/expected/oidjoins.out | 4 + src/tools/pgindent/typedefs.list | 5 + 41 files changed, 1588 insertions(+), 12 deletions(-) create mode 100644 src/backend/catalog/pg_variable.c create mode 100644 src/backend/commands/session_variable.c create mode 100644 src/include/catalog/pg_variable.h create mode 100644 src/include/commands/session_variable.h diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 6a837e1539..ee20c2e520 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -36,6 +36,7 @@ #include "catalog/pg_enum.h" #include "catalog/storage.h" #include "commands/async.h" +#include "commands/session_variable.h" #include "commands/tablecmds.h" #include "commands/trigger.h" #include "common/pg_prng.h" @@ -2224,6 +2225,9 @@ CommitTransaction(void) */ smgrDoPendingSyncs(true, is_parallel_worker); + /* Let ON COMMIT DROP */ + AtPreEOXact_SessionVariable(true); + /* close large objects before lower-level cleanup */ AtEOXact_LargeObject(true); @@ -2800,6 +2804,9 @@ AbortTransaction(void) AtAbort_Portals(); smgrDoPendingSyncs(false, is_parallel_worker); AtEOXact_LargeObject(false); + + /* 'false' means it's abort */ + AtPreEOXact_SessionVariable(false); AtAbort_Notify(); AtEOXact_RelationMap(false, is_parallel_worker); AtAbort_Twophase(); @@ -5024,6 +5031,8 @@ CommitSubTransaction(void) AtEOSubXact_SPI(true, s->subTransactionId); AtEOSubXact_on_commit_actions(true, s->subTransactionId, s->parent->subTransactionId); + AtEOSubXact_SessionVariable(true, s->subTransactionId, + s->parent->subTransactionId); AtEOSubXact_Namespace(true, s->subTransactionId, s->parent->subTransactionId); AtEOSubXact_Files(true, s->subTransactionId, @@ -5187,6 +5196,8 @@ AbortSubTransaction(void) AtEOSubXact_SPI(false, s->subTransactionId); AtEOSubXact_on_commit_actions(false, s->subTransactionId, s->parent->subTransactionId); + AtEOSubXact_SessionVariable(false, s->subTransactionId, + s->parent->subTransactionId); AtEOSubXact_Namespace(false, s->subTransactionId, s->parent->subTransactionId); AtEOSubXact_Files(false, s->subTransactionId, diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile index a60107bf94..34a8ac98a8 100644 --- a/src/backend/catalog/Makefile +++ b/src/backend/catalog/Makefile @@ -45,6 +45,7 @@ OBJS = \ pg_shdepend.o \ pg_subscription.o \ pg_type.o \ + pg_variable.o \ storage.o \ toasting.o @@ -72,7 +73,8 @@ CATALOG_HEADERS := \ pg_collation.h pg_parameter_acl.h pg_partitioned_table.h \ pg_range.h pg_transform.h \ pg_sequence.h pg_publication.h pg_publication_namespace.h \ - pg_publication_rel.h pg_subscription.h pg_subscription_rel.h + pg_publication_rel.h pg_subscription.h pg_subscription_rel.h \ + pg_variable.h GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h) schemapg.h system_fk_info.h diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c index 45cdcd3dc6..c2f6ca4a1d 100644 --- a/src/backend/catalog/aclchk.c +++ b/src/backend/catalog/aclchk.c @@ -60,6 +60,7 @@ #include "catalog/pg_ts_parser.h" #include "catalog/pg_ts_template.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "commands/dbcommands.h" #include "commands/defrem.h" #include "commands/event_trigger.h" @@ -281,6 +282,9 @@ restrict_and_check_grant(bool is_grant, AclMode avail_goptions, bool all_privs, case OBJECT_PARAMETER_ACL: whole_mask = ACL_ALL_RIGHTS_PARAMETER_ACL; break; + case OBJECT_VARIABLE: + whole_mask = ACL_ALL_RIGHTS_VARIABLE; + break; default: elog(ERROR, "unrecognized object type: %d", objtype); /* not reached, but keep compiler quiet */ @@ -525,6 +529,10 @@ ExecuteGrantStmt(GrantStmt *stmt) all_privileges = ACL_ALL_RIGHTS_PARAMETER_ACL; errormsg = gettext_noop("invalid privilege type %s for parameter"); break; + case OBJECT_VARIABLE: + all_privileges = ACL_ALL_RIGHTS_VARIABLE; + errormsg = gettext_noop("invalid privilege type %s for session variable"); + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) stmt->objtype); @@ -630,6 +638,9 @@ ExecGrantStmt_oids(InternalGrant *istmt) case OBJECT_PARAMETER_ACL: ExecGrant_Parameter(istmt); break; + case OBJECT_VARIABLE: + ExecGrant_common(istmt, VariableRelationId, ACL_ALL_RIGHTS_VARIABLE, NULL); + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) istmt->objtype); @@ -820,6 +831,18 @@ objectNamesToOids(ObjectType objtype, List *objnames, bool is_grant) objects = lappend_oid(objects, parameterId); } break; + case OBJECT_VARIABLE: + foreach(cell, objnames) + { + RangeVar *varvar = (RangeVar *) lfirst(cell); + Oid relOid; + + relOid = LookupVariable(varvar->schemaname, + varvar->relname, + false); + objects = lappend_oid(objects, relOid); + } + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) objtype); @@ -909,6 +932,32 @@ objectsInSchemaToOids(ObjectType objtype, List *nspnames) table_close(rel, AccessShareLock); } break; + case OBJECT_VARIABLE: + { + ScanKeyData key; + Relation rel; + TableScanDesc scan; + HeapTuple tuple; + + ScanKeyInit(&key, + Anum_pg_variable_varnamespace, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(namespaceId)); + + rel = table_open(VariableRelationId, AccessShareLock); + scan = table_beginscan_catalog(rel, 1, &key); + + while ((tuple = heap_getnext(scan, ForwardScanDirection)) != NULL) + { + Oid oid = ((Form_pg_proc) GETSTRUCT(tuple))->oid; + + objects = lappend_oid(objects, oid); + } + + table_endscan(scan); + table_close(rel, AccessShareLock); + } + break; default: /* should not happen */ elog(ERROR, "unrecognized GrantStmt.objtype: %d", @@ -1068,6 +1117,10 @@ ExecAlterDefaultPrivilegesStmt(ParseState *pstate, AlterDefaultPrivilegesStmt *s all_privileges = ACL_ALL_RIGHTS_SCHEMA; errormsg = gettext_noop("invalid privilege type %s for schema"); break; + case OBJECT_VARIABLE: + all_privileges = ACL_ALL_RIGHTS_VARIABLE; + errormsg = gettext_noop("invalid privilege type %s for session variable"); + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) action->objtype); @@ -1259,6 +1312,12 @@ SetDefaultACL(InternalDefaultACL *iacls) this_privileges = ACL_ALL_RIGHTS_SCHEMA; break; + case OBJECT_VARIABLE: + objtype = DEFACLOBJ_VARIABLE; + if (iacls->all_privs && this_privileges == ACL_NO_RIGHTS) + this_privileges = ACL_ALL_RIGHTS_VARIABLE; + break; + default: elog(ERROR, "unrecognized object type: %d", (int) iacls->objtype); @@ -1490,6 +1549,9 @@ RemoveRoleFromObjectACL(Oid roleid, Oid classid, Oid objid) case DEFACLOBJ_NAMESPACE: iacls.objtype = OBJECT_SCHEMA; break; + case DEFACLOBJ_VARIABLE: + iacls.objtype = OBJECT_VARIABLE; + break; default: /* Shouldn't get here */ elog(ERROR, "unexpected default ACL type: %d", @@ -1550,6 +1612,9 @@ RemoveRoleFromObjectACL(Oid roleid, Oid classid, Oid objid) case ParameterAclRelationId: istmt.objtype = OBJECT_PARAMETER_ACL; break; + case VariableRelationId: + istmt.objtype = OBJECT_VARIABLE; + break; default: elog(ERROR, "unexpected object class %u", classid); break; @@ -2783,6 +2848,9 @@ aclcheck_error(AclResult aclerr, ObjectType objtype, case OBJECT_TYPE: msg = gettext_noop("permission denied for type %s"); break; + case OBJECT_VARIABLE: + msg = gettext_noop("permission denied for session variable %s"); + break; case OBJECT_VIEW: msg = gettext_noop("permission denied for view %s"); break; @@ -2894,6 +2962,9 @@ aclcheck_error(AclResult aclerr, ObjectType objtype, case OBJECT_TYPE: msg = gettext_noop("must be owner of type %s"); break; + case OBJECT_VARIABLE: + msg = gettext_noop("must be owner of session variable %s"); + break; case OBJECT_VIEW: msg = gettext_noop("must be owner of view %s"); break; @@ -3042,6 +3113,8 @@ pg_aclmask(ObjectType objtype, Oid object_oid, AttrNumber attnum, Oid roleid, return ACL_NO_RIGHTS; case OBJECT_TYPE: return object_aclmask(TypeRelationId, object_oid, roleid, mask, how); + case OBJECT_VARIABLE: + return object_aclmask(VariableRelationId, object_oid, roleid, mask, how); default: elog(ERROR, "unrecognized object type: %d", (int) objtype); @@ -4166,6 +4239,10 @@ get_user_default_acl(ObjectType objtype, Oid ownerId, Oid nsp_oid) defaclobjtype = DEFACLOBJ_NAMESPACE; break; + case OBJECT_VARIABLE: + defaclobjtype = DEFACLOBJ_VARIABLE; + break; + default: return NULL; } diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index f8a136ba0a..77acbeda80 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -65,12 +65,15 @@ #include "catalog/pg_ts_template.h" #include "catalog/pg_type.h" #include "catalog/pg_user_mapping.h" +#include "catalog/pg_variable.h" #include "commands/comment.h" #include "commands/defrem.h" #include "commands/event_trigger.h" #include "commands/extension.h" #include "commands/policy.h" #include "commands/publicationcmds.h" +#include "commands/schemacmds.h" +#include "commands/session_variable.h" #include "commands/seclabel.h" #include "commands/sequence.h" #include "commands/trigger.h" @@ -188,7 +191,8 @@ static const Oid object_classes[] = { PublicationRelationId, /* OCLASS_PUBLICATION */ PublicationRelRelationId, /* OCLASS_PUBLICATION_REL */ SubscriptionRelationId, /* OCLASS_SUBSCRIPTION */ - TransformRelationId /* OCLASS_TRANSFORM */ + TransformRelationId, /* OCLASS_TRANSFORM */ + VariableRelationId /* OCLASS_VARIABLE */ }; /* @@ -1514,6 +1518,10 @@ doDeletion(const ObjectAddress *object, int flags) DropObjectById(object); break; + case OCLASS_VARIABLE: + DropVariable(object->objectId); + break; + /* * These global object types are not supported here. */ @@ -2966,6 +2974,9 @@ getObjectClass(const ObjectAddress *object) case TransformRelationId: return OCLASS_TRANSFORM; + + case VariableRelationId: + return OCLASS_VARIABLE; } /* shouldn't get here */ diff --git a/src/backend/catalog/meson.build b/src/backend/catalog/meson.build index fa6609e577..f48e3cb07d 100644 --- a/src/backend/catalog/meson.build +++ b/src/backend/catalog/meson.build @@ -32,6 +32,7 @@ backend_sources += files( 'pg_shdepend.c', 'pg_subscription.c', 'pg_type.c', + 'pg_variable.c', 'storage.c', 'toasting.c', ) diff --git a/src/backend/catalog/namespace.c b/src/backend/catalog/namespace.c index 73ddb67882..9a04530d1e 100644 --- a/src/backend/catalog/namespace.c +++ b/src/backend/catalog/namespace.c @@ -40,6 +40,7 @@ #include "catalog/pg_ts_parser.h" #include "catalog/pg_ts_template.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "commands/dbcommands.h" #include "funcapi.h" #include "mb/pg_wchar.h" @@ -765,6 +766,69 @@ RelationIsVisible(Oid relid) return visible; } +/* + * VariableIsVisible + * Determine whether a variable (identified by OID) is visible in the + * current search path. Visible means "would be found by searching + * for the unqualified variable name". + */ +bool +VariableIsVisible(Oid varid) +{ + HeapTuple vartup; + Form_pg_variable varform; + Oid varnamespace; + bool visible; + + vartup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + if (!HeapTupleIsValid(vartup)) + elog(ERROR, "cache lookup failed for session variable %u", varid); + varform = (Form_pg_variable) GETSTRUCT(vartup); + + recomputeNamespacePath(); + + /* + * Quick check: if it ain't in the path at all, it ain't visible. Items in + * the system namespace are surely in the path and so we needn't even do + * list_member_oid() for them. + */ + varnamespace = varform->varnamespace; + if (varnamespace != PG_CATALOG_NAMESPACE && + !list_member_oid(activeSearchPath, varnamespace)) + visible = false; + else + { + /* + * If it is in the path, it might still not be visible; it could be + * hidden by another variable of the same name earlier in the path. So + * we must do a slow check for conflicting relations. + */ + char *varname = NameStr(varform->varname); + ListCell *l; + + visible = false; + foreach(l, activeSearchPath) + { + Oid namespaceId = lfirst_oid(l); + + if (namespaceId == varnamespace) + { + /* Found it first in path */ + visible = true; + break; + } + if (OidIsValid(get_varname_varid(varname, namespaceId))) + { + /* Found something else first in path */ + break; + } + } + } + + ReleaseSysCache(vartup); + + return visible; +} /* * TypenameGetTypid @@ -2840,6 +2904,71 @@ TSConfigIsVisible(Oid cfgid) return visible; } +/* + * Returns oid of session variable specified by possibly qualified identifier. + * + * If not found, returns InvalidOid if missing_ok, else throws error. + * When rowtype_only argument is true the session variables of not + * composite types are ignored. This should to reduce possible collisions. + */ +Oid +LookupVariable(const char *nspname, + const char *varname, + bool missing_ok) +{ + Oid namespaceId; + Oid varoid = InvalidOid; + ListCell *l; + + if (nspname) + { + namespaceId = LookupExplicitNamespace(nspname, missing_ok); + + /* + * If nspname is not a known namespace, then nspname.varname cannot be + * any usable session variable. + */ + if (OidIsValid(namespaceId)) + { + varoid = GetSysCacheOid2(VARIABLENAMENSP, Anum_pg_variable_oid, + PointerGetDatum(varname), + ObjectIdGetDatum(namespaceId)); + } + } + else + { + /* Iterate over schemas in search_path */ + recomputeNamespacePath(); + + foreach(l, activeSearchPath) + { + namespaceId = lfirst_oid(l); + + varoid = GetSysCacheOid2(VARIABLENAMENSP, Anum_pg_variable_oid, + PointerGetDatum(varname), + ObjectIdGetDatum(namespaceId)); + + if (OidIsValid(varoid)) + break; + } + } + + if (!OidIsValid(varoid) && !missing_ok) + { + if (nspname) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("session variable \"%s.%s\" does not exist", + nspname, varname))); + else + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("session variable \"%s\" does not exist", + varname))); + } + + return varoid; +} /* * DeconstructQualifiedName @@ -4661,3 +4790,14 @@ pg_is_other_temp_schema(PG_FUNCTION_ARGS) PG_RETURN_BOOL(isOtherTempNamespace(oid)); } + +Datum +pg_variable_is_visible(PG_FUNCTION_ARGS) +{ + Oid oid = PG_GETARG_OID(0); + + if (!SearchSysCacheExists1(VARIABLEOID, ObjectIdGetDatum(oid))) + PG_RETURN_NULL(); + + PG_RETURN_BOOL(VariableIsVisible(oid)); +} diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c index 95fefc7565..d5930f2627 100644 --- a/src/backend/catalog/objectaddress.c +++ b/src/backend/catalog/objectaddress.c @@ -64,6 +64,7 @@ #include "catalog/pg_ts_template.h" #include "catalog/pg_type.h" #include "catalog/pg_user_mapping.h" +#include "catalog/pg_variable.h" #include "commands/dbcommands.h" #include "commands/defrem.h" #include "commands/event_trigger.h" @@ -633,6 +634,20 @@ static const ObjectPropertyType ObjectProperty[] = OBJECT_USER_MAPPING, false }, + { + "session variable", + VariableRelationId, + VariableObjectIndexId, + VARIABLEOID, + VARIABLENAMENSP, + Anum_pg_variable_oid, + Anum_pg_variable_varname, + Anum_pg_variable_varnamespace, + Anum_pg_variable_varowner, + Anum_pg_variable_varacl, + OBJECT_VARIABLE, + true + } }; /* @@ -869,6 +884,10 @@ static const struct object_type_map /* OCLASS_STATISTIC_EXT */ { "statistics object", OBJECT_STATISTIC_EXT + }, + /* OCLASS_VARIABLE */ + { + "session variable", OBJECT_VARIABLE } }; @@ -894,6 +913,7 @@ static ObjectAddress get_object_address_attrdef(ObjectType objtype, bool missing_ok); static ObjectAddress get_object_address_type(ObjectType objtype, TypeName *typename, bool missing_ok); +static ObjectAddress get_object_address_variable(List *object, bool missing_ok); static ObjectAddress get_object_address_opcf(ObjectType objtype, List *object, bool missing_ok); static ObjectAddress get_object_address_opf_member(ObjectType objtype, @@ -1164,6 +1184,9 @@ get_object_address(ObjectType objtype, Node *object, missing_ok); address.objectSubId = 0; break; + case OBJECT_VARIABLE: + address = get_object_address_variable(castNode(List, object), missing_ok); + break; /* no default, to let compiler warn about missing case */ } @@ -2038,16 +2061,20 @@ get_object_address_defacl(List *object, bool missing_ok) case DEFACLOBJ_NAMESPACE: objtype_str = "schemas"; break; + case DEFACLOBJ_VARIABLE: + objtype_str = "variables"; + break; default: ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("unrecognized default ACL object type \"%c\"", objtype), - errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", + errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", DEFACLOBJ_RELATION, DEFACLOBJ_SEQUENCE, DEFACLOBJ_FUNCTION, DEFACLOBJ_TYPE, - DEFACLOBJ_NAMESPACE))); + DEFACLOBJ_NAMESPACE, + DEFACLOBJ_VARIABLE))); } /* @@ -2131,6 +2158,24 @@ textarray_to_strvaluelist(ArrayType *arr) return list; } +/* + * Find the ObjectAddress for a session variable + */ +static ObjectAddress +get_object_address_variable(List *object, bool missing_ok) +{ + ObjectAddress address; + char *nspname = NULL; + char *varname = NULL; + + ObjectAddressSet(address, VariableRelationId, InvalidOid); + + DeconstructQualifiedName(object, &nspname, &varname); + address.objectId = LookupVariable(nspname, varname, missing_ok); + + return address; +} + /* * SQL-callable version of get_object_address */ @@ -2325,6 +2370,7 @@ pg_get_object_address(PG_FUNCTION_ARGS) case OBJECT_TABCONSTRAINT: case OBJECT_OPCLASS: case OBJECT_OPFAMILY: + case OBJECT_VARIABLE: objnode = (Node *) name; break; case OBJECT_ACCESS_METHOD: @@ -2496,6 +2542,7 @@ check_object_ownership(Oid roleid, ObjectType objtype, ObjectAddress address, case OBJECT_STATISTIC_EXT: case OBJECT_TSDICTIONARY: case OBJECT_TSCONFIGURATION: + case OBJECT_VARIABLE: if (!object_ownercheck(address.classId, address.objectId, roleid)) aclcheck_error(ACLCHECK_NOT_OWNER, objtype, NameListToString(castNode(List, object))); @@ -3484,6 +3531,32 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok) break; } + case OCLASS_VARIABLE: + { + char *nspname; + HeapTuple tup; + Form_pg_variable varform; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(object->objectId)); + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for session variable %u", + object->objectId); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + if (VariableIsVisible(object->objectId)) + nspname = NULL; + else + nspname = get_namespace_name(varform->varnamespace); + + appendStringInfo(&buffer, _("session variable %s"), + quote_qualified_identifier(nspname, + NameStr(varform->varname))); + + ReleaseSysCache(tup); + break; + } + case OCLASS_TSPARSER: { HeapTuple tup; @@ -3836,6 +3909,16 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok) _("default privileges on new schemas belonging to role %s"), rolename); break; + case DEFACLOBJ_VARIABLE: + if (nspname) + appendStringInfo(&buffer, + _("default privileges on new session variables belonging to role %s in schema %s"), + rolename, nspname); + else + appendStringInfo(&buffer, + _("default privileges on new session variables belonging to role %s"), + rolename); + break; default: /* shouldn't get here */ if (nspname) @@ -4585,6 +4668,10 @@ getObjectTypeDescription(const ObjectAddress *object, bool missing_ok) appendStringInfoString(&buffer, "transform"); break; + case OCLASS_VARIABLE: + appendStringInfoString(&buffer, "session variable"); + break; + /* * There's intentionally no default: case here; we want the * compiler to warn if a new OCLASS hasn't been handled above. @@ -5692,6 +5779,10 @@ getObjectIdentityParts(const ObjectAddress *object, appendStringInfoString(&buffer, " on schemas"); break; + case DEFACLOBJ_VARIABLE: + appendStringInfoString(&buffer, + " on session variables"); + break; } if (objname) @@ -5932,6 +6023,33 @@ getObjectIdentityParts(const ObjectAddress *object, } break; + case OCLASS_VARIABLE: + { + char *schema; + char *varname; + HeapTuple tup; + Form_pg_variable varform; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(object->objectId)); + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for session variable %u", + object->objectId); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + schema = get_namespace_name_or_temp(varform->varnamespace); + varname = NameStr(varform->varname); + + appendStringInfo(&buffer, "%s", + quote_qualified_identifier(schema, varname)); + + if (objname) + *objname = list_make2(schema, varname); + + ReleaseSysCache(tup); + break; + } + /* * There's intentionally no default: case here; we want the * compiler to warn if a new OCLASS hasn't been handled above. diff --git a/src/backend/catalog/pg_shdepend.c b/src/backend/catalog/pg_shdepend.c index 64d326f073..848b36a87e 100644 --- a/src/backend/catalog/pg_shdepend.c +++ b/src/backend/catalog/pg_shdepend.c @@ -47,6 +47,7 @@ #include "catalog/pg_ts_dict.h" #include "catalog/pg_type.h" #include "catalog/pg_user_mapping.h" +#include "catalog/pg_variable.h" #include "commands/alter.h" #include "commands/collationcmds.h" #include "commands/conversioncmds.h" @@ -1613,6 +1614,7 @@ shdepReassignOwned(List *roleids, Oid newrole) case DatabaseRelationId: case TSConfigRelationId: case TSDictionaryRelationId: + case VariableRelationId: { Oid classId = sdepForm->classid; Relation catalog; diff --git a/src/backend/catalog/pg_variable.c b/src/backend/catalog/pg_variable.c new file mode 100644 index 0000000000..462126bf7a --- /dev/null +++ b/src/backend/catalog/pg_variable.c @@ -0,0 +1,378 @@ +/*------------------------------------------------------------------------- + * + * pg_variable.c + * session variables + * + * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/backend/catalog/pg_variable.c + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/heapam.h" +#include "access/htup_details.h" +#include "catalog/catalog.h" +#include "catalog/dependency.h" +#include "catalog/indexing.h" +#include "catalog/namespace.h" +#include "catalog/objectaccess.h" +#include "catalog/pg_collation.h" +#include "catalog/pg_namespace.h" +#include "catalog/pg_type.h" +#include "catalog/pg_variable.h" +#include "commands/session_variable.h" +#include "miscadmin.h" +#include "parser/parse_coerce.h" +#include "parser/parse_collate.h" +#include "parser/parse_expr.h" +#include "parser/parse_type.h" +#include "storage/lmgr.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/pg_lsn.h" +#include "utils/syscache.h" + + +static ObjectAddress create_variable(const char *varName, + Oid varNamespace, + Oid varType, + int32 varTypmod, + Oid varOwner, + Oid varCollation, + Node *varDefexpr, + VariableEOXAction eoxaction, + bool is_not_null, + bool if_not_exists, + bool is_immutable); + + +/* + * Creates entry in pg_variable table + */ +static ObjectAddress +create_variable(const char *varName, + Oid varNamespace, + Oid varType, + int32 varTypmod, + Oid varOwner, + Oid varCollation, + Node *varDefexpr, + VariableEOXAction eoxaction, + bool is_not_null, + bool if_not_exists, + bool is_immutable) +{ + Acl *varacl; + NameData varname; + bool nulls[Natts_pg_variable]; + Datum values[Natts_pg_variable]; + Relation rel; + HeapTuple tup; + TupleDesc tupdesc; + ObjectAddress myself, + referenced; + ObjectAddresses *addrs; + Oid varid; + + Assert(varName); + Assert(OidIsValid(varNamespace)); + Assert(OidIsValid(varType)); + Assert(OidIsValid(varOwner)); + + rel = table_open(VariableRelationId, RowExclusiveLock); + + /* + * Check for duplicates. Note that this does not really prevent + * duplicates, it's here just to provide nicer error message in common + * case. The real protection is the unique key on the catalog. + */ + if (SearchSysCacheExists2(VARIABLENAMENSP, + PointerGetDatum(varName), + ObjectIdGetDatum(varNamespace))) + { + if (if_not_exists) + ereport(NOTICE, + (errcode(ERRCODE_DUPLICATE_OBJECT), + errmsg("session variable \"%s\" already exists, skipping", + varName))); + else + ereport(ERROR, + (errcode(ERRCODE_DUPLICATE_OBJECT), + errmsg("session variable \"%s\" already exists", + varName))); + + table_close(rel, RowExclusiveLock); + + return InvalidObjectAddress; + } + + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + + namestrcpy(&varname, varName); + + varid = GetNewOidWithIndex(rel, VariableObjectIndexId, Anum_pg_variable_oid); + + values[Anum_pg_variable_oid - 1] = ObjectIdGetDatum(varid); + values[Anum_pg_variable_varcreate_lsn - 1] = LSNGetDatum(GetXLogInsertRecPtr()); + values[Anum_pg_variable_varname - 1] = NameGetDatum(&varname); + values[Anum_pg_variable_varnamespace - 1] = ObjectIdGetDatum(varNamespace); + values[Anum_pg_variable_vartype - 1] = ObjectIdGetDatum(varType); + values[Anum_pg_variable_vartypmod - 1] = Int32GetDatum(varTypmod); + values[Anum_pg_variable_varowner - 1] = ObjectIdGetDatum(varOwner); + values[Anum_pg_variable_varcollation - 1] = ObjectIdGetDatum(varCollation); + values[Anum_pg_variable_varisnotnull - 1] = BoolGetDatum(is_not_null); + values[Anum_pg_variable_varisimmutable - 1] = BoolGetDatum(is_immutable); + values[Anum_pg_variable_vareoxaction - 1] = CharGetDatum(eoxaction); + + /* varacl will be determined later */ + + if (varDefexpr) + values[Anum_pg_variable_vardefexpr - 1] = CStringGetTextDatum(nodeToString(varDefexpr)); + else + nulls[Anum_pg_variable_vardefexpr - 1] = true; + + tupdesc = RelationGetDescr(rel); + + varacl = get_user_default_acl(OBJECT_VARIABLE, varOwner, + varNamespace); + + if (varacl != NULL) + values[Anum_pg_variable_varacl - 1] = PointerGetDatum(varacl); + else + nulls[Anum_pg_variable_varacl - 1] = true; + + tup = heap_form_tuple(tupdesc, values, nulls); + CatalogTupleInsert(rel, tup); + Assert(OidIsValid(varid)); + + addrs = new_object_addresses(); + + ObjectAddressSet(myself, VariableRelationId, varid); + + /* dependency on namespace */ + ObjectAddressSet(referenced, NamespaceRelationId, varNamespace); + add_exact_object_address(&referenced, addrs); + + /* dependency on used type */ + ObjectAddressSet(referenced, TypeRelationId, varType); + add_exact_object_address(&referenced, addrs); + + /* dependency on collation */ + if (OidIsValid(varCollation) && + varCollation != DEFAULT_COLLATION_OID) + { + ObjectAddressSet(referenced, CollationRelationId, varCollation); + add_exact_object_address(&referenced, addrs); + } + + /* dependency on default expr */ + if (varDefexpr) + recordDependencyOnExpr(&myself, (Node *) varDefexpr, + NIL, DEPENDENCY_NORMAL); + + record_object_address_dependencies(&myself, addrs, DEPENDENCY_NORMAL); + free_object_addresses(addrs); + + /* dependency on owner */ + recordDependencyOnOwner(VariableRelationId, varid, varOwner); + + /* dependencies on roles mentioned in default ACL */ + recordDependencyOnNewAcl(VariableRelationId, varid, 0, varOwner, varacl); + + /* dependency on extension */ + recordDependencyOnCurrentExtension(&myself, false); + + heap_freetuple(tup); + + /* Post creation hook for new function */ + InvokeObjectPostCreateHook(VariableRelationId, varid, 0); + + table_close(rel, RowExclusiveLock); + + return myself; +} + +/* + * Creates a new variable + * + * Used by CREATE VARIABLE command + */ +ObjectAddress +CreateVariable(ParseState *pstate, CreateSessionVarStmt *stmt) +{ + Oid namespaceid; + AclResult aclresult; + Oid typid; + int32 typmod; + Oid varowner = GetUserId(); + Oid collation; + Oid typcollation; + ObjectAddress variable; + + Node *cooked_default = NULL; + + /* Check consistency of arguments */ + if (stmt->eoxaction == VARIABLE_EOX_DROP + && stmt->variable->relpersistence != RELPERSISTENCE_TEMP) + ereport(ERROR, + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), + errmsg("ON COMMIT DROP can only be used on temporary variables"))); + + if (stmt->is_not_null && stmt->is_immutable && !stmt->defexpr) + ereport(ERROR, + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), + errmsg("IMMUTABLE NOT NULL variable requires default expression"))); + + namespaceid = + RangeVarGetAndCheckCreationNamespace(stmt->variable, NoLock, NULL); + + typenameTypeIdAndMod(pstate, stmt->typeName, &typid, &typmod); + + /* Disallow pseudotypes */ + if (get_typtype(typid) == TYPTYPE_PSEUDO) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("session variable cannot be pseudo-type %s", + format_type_be(typid)))); + + aclresult = object_aclcheck(TypeRelationId, typid, GetUserId(), ACL_USAGE); + if (aclresult != ACLCHECK_OK) + aclcheck_error_type(aclresult, typid); + + typcollation = get_typcollation(typid); + + if (stmt->collClause) + collation = LookupCollation(pstate, + stmt->collClause->collname, + stmt->collClause->location); + else + collation = typcollation;; + + /* Complain if COLLATE is applied to an uncollatable type */ + if (OidIsValid(collation) && !OidIsValid(typcollation)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("collations are not supported by type %s", + format_type_be(typid)), + parser_errposition(pstate, stmt->collClause->location))); + + if (stmt->defexpr) + { + cooked_default = transformExpr(pstate, stmt->defexpr, + EXPR_KIND_VARIABLE_DEFAULT); + + cooked_default = coerce_to_specific_type(pstate, + cooked_default, typid, "DEFAULT"); + assign_expr_collations(pstate, cooked_default); + } + + variable = create_variable(stmt->variable->relname, + namespaceid, + typid, + typmod, + varowner, + collation, + cooked_default, + stmt->eoxaction, + stmt->is_not_null, + stmt->if_not_exists, + stmt->is_immutable); + + elog(DEBUG1, "record for session variable \"%s\" (oid:%d) was created in pg_variable", + stmt->variable->relname, variable.objectId); + + /* We want SessionVariableCreatePostprocess to see the catalog changes. */ + CommandCounterIncrement(); + + SessionVariableCreatePostprocess(variable.objectId, stmt->eoxaction); + + return variable; +} + +/* + * Drop variable by OID, and register the needed session variable + * cleanup. + */ +void +DropVariable(Oid varid) +{ + Relation rel; + HeapTuple tup; + + rel = table_open(VariableRelationId, RowExclusiveLock); + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for variable %u", varid); + + CatalogTupleDelete(rel, &tup->t_self); + + ReleaseSysCache(tup); + + table_close(rel, RowExclusiveLock); + + /* Do the necessary cleanup if needed in local memory */ + SessionVariableDropPostprocess(varid); +} + +/* + * Fetch attributes (without acl) of session variable from the syscache. + * We don't work with acl directly, so we don't need to read it here. + * Skip deserialization of defexpr when fast_only is true. + */ +void +InitVariable(Variable *var, Oid varid, bool fast_only) +{ + HeapTuple tup; + Form_pg_variable varform; + Datum defexpr_datum; + bool defexpr_isnull; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for session variable %u", varid); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + var->oid = varid; + var->create_lsn = varform->varcreate_lsn; + var->name = pstrdup(NameStr(varform->varname)); + var->namespaceid = varform->varnamespace; + var->typid = varform->vartype; + var->typmod = varform->vartypmod; + var->owner = varform->varowner; + var->collation = varform->varcollation; + var->is_immutable = varform->varisimmutable; + var->is_not_null = varform->varisnotnull; + var->eoxaction = varform->vareoxaction; + + /* Get defexpr */ + defexpr_datum = SysCacheGetAttr(VARIABLEOID, + tup, + Anum_pg_variable_vardefexpr, + &defexpr_isnull); + + var->has_defexpr = !defexpr_isnull; + + /* + * Deserialize defexpr only when it is requested. We need to deserialize + * Node with default expression, only when we read from session variable, + * and this session variable has not assigned value, and this session + * variable has default expression. For other cases, we skip skip this + * operation. + */ + if (!fast_only && !defexpr_isnull) + var->defexpr = stringToNode(TextDatumGetCString(defexpr_datum)); + else + var->defexpr = NULL; + + ReleaseSysCache(tup); +} diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile index 48f7348f91..1cfaeca51e 100644 --- a/src/backend/commands/Makefile +++ b/src/backend/commands/Makefile @@ -50,6 +50,7 @@ OBJS = \ schemacmds.o \ seclabel.o \ sequence.o \ + session_variable.o \ statscmds.o \ subscriptioncmds.o \ tablecmds.o \ diff --git a/src/backend/commands/alter.c b/src/backend/commands/alter.c index 10f28f94bc..9ca83beb1c 100644 --- a/src/backend/commands/alter.c +++ b/src/backend/commands/alter.c @@ -41,6 +41,7 @@ #include "catalog/pg_ts_dict.h" #include "catalog/pg_ts_parser.h" #include "catalog/pg_ts_template.h" +#include "catalog/pg_variable.h" #include "commands/alter.h" #include "commands/collationcmds.h" #include "commands/conversioncmds.h" @@ -143,6 +144,10 @@ report_namespace_conflict(Oid classId, const char *name, Oid nspOid) Assert(OidIsValid(nspOid)); msgfmt = gettext_noop("text search configuration \"%s\" already exists in schema \"%s\""); break; + case VariableRelationId: + Assert(OidIsValid(nspOid)); + msgfmt = gettext_noop("session variable \"%s\" already exists in schema \"%s\""); + break; default: elog(ERROR, "unsupported object class: %u", classId); break; @@ -421,6 +426,7 @@ ExecRenameStmt(RenameStmt *stmt) case OBJECT_TSTEMPLATE: case OBJECT_PUBLICATION: case OBJECT_SUBSCRIPTION: + case OBJECT_VARIABLE: { ObjectAddress address; Relation catalog; @@ -564,6 +570,7 @@ ExecAlterObjectSchemaStmt(AlterObjectSchemaStmt *stmt, case OBJECT_TSDICTIONARY: case OBJECT_TSPARSER: case OBJECT_TSTEMPLATE: + case OBJECT_VARIABLE: { Relation catalog; Relation relation; @@ -654,6 +661,7 @@ AlterObjectNamespace_oid(Oid classId, Oid objid, Oid nspOid, case OCLASS_TSDICT: case OCLASS_TSTEMPLATE: case OCLASS_TSCONFIG: + case OCLASS_VARIABLE: { Relation catalog; @@ -914,6 +922,7 @@ ExecAlterOwnerStmt(AlterOwnerStmt *stmt) case OBJECT_TABLESPACE: case OBJECT_TSDICTIONARY: case OBJECT_TSCONFIGURATION: + case OBJECT_VARIABLE: { Relation catalog; Relation relation; diff --git a/src/backend/commands/dropcmds.c b/src/backend/commands/dropcmds.c index 82bda15889..a5423b687e 100644 --- a/src/backend/commands/dropcmds.c +++ b/src/backend/commands/dropcmds.c @@ -481,6 +481,10 @@ does_not_exist_skipping(ObjectType objtype, Node *object) msg = gettext_noop("publication \"%s\" does not exist, skipping"); name = strVal(object); break; + case OBJECT_VARIABLE: + msg = gettext_noop("session variable \"%s\" does not exist, skipping"); + name = NameListToString(castNode(List, object)); + break; case OBJECT_COLUMN: case OBJECT_DATABASE: diff --git a/src/backend/commands/event_trigger.c b/src/backend/commands/event_trigger.c index d4b00d1a82..ed2a9d776b 100644 --- a/src/backend/commands/event_trigger.c +++ b/src/backend/commands/event_trigger.c @@ -991,6 +991,7 @@ EventTriggerSupportsObjectType(ObjectType obtype) case OBJECT_TSTEMPLATE: case OBJECT_TYPE: case OBJECT_USER_MAPPING: + case OBJECT_VARIABLE: case OBJECT_VIEW: return true; @@ -1057,6 +1058,7 @@ EventTriggerSupportsObjectClass(ObjectClass objclass) case OCLASS_PUBLICATION_REL: case OCLASS_SUBSCRIPTION: case OCLASS_TRANSFORM: + case OCLASS_VARIABLE: return true; /* @@ -2049,6 +2051,8 @@ stringify_grant_objtype(ObjectType objtype) return "TABLESPACE"; case OBJECT_TYPE: return "TYPE"; + case OBJECT_VARIABLE: + return "VARIABLE"; /* these currently aren't used */ case OBJECT_ACCESS_METHOD: case OBJECT_AGGREGATE: @@ -2132,6 +2136,8 @@ stringify_adefprivs_objtype(ObjectType objtype) return "TABLESPACES"; case OBJECT_TYPE: return "TYPES"; + case OBJECT_VARIABLE: + return "VARIABLES"; /* these currently aren't used */ case OBJECT_ACCESS_METHOD: case OBJECT_AGGREGATE: diff --git a/src/backend/commands/seclabel.c b/src/backend/commands/seclabel.c index 7ff16e3276..75b39b2945 100644 --- a/src/backend/commands/seclabel.c +++ b/src/backend/commands/seclabel.c @@ -92,6 +92,7 @@ SecLabelSupportsObjectType(ObjectType objtype) case OBJECT_TSPARSER: case OBJECT_TSTEMPLATE: case OBJECT_USER_MAPPING: + case OBJECT_VARIABLE: return false; /* diff --git a/src/backend/commands/session_variable.c b/src/backend/commands/session_variable.c new file mode 100644 index 0000000000..946e73e467 --- /dev/null +++ b/src/backend/commands/session_variable.c @@ -0,0 +1,243 @@ +/*------------------------------------------------------------------------- + * + * session_variable.c + * session variable creation/manipulation commands + * + * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/commands/sessionvariable.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/heapam.h" +#include "catalog/dependency.h" +#include "catalog/indexing.h" +#include "catalog/namespace.h" +#include "catalog/pg_variable.h" +#include "commands/session_variable.h" +#include "funcapi.h" +#include "miscadmin.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/memutils.h" +#include "utils/syscache.h" + +/* + * The life cycle of temporary session variable can be + * limmited by using clause ON COMMIT DROP. + */ +typedef enum SVariableXActAction +{ + SVAR_ON_COMMIT_DROP, /* used for ON COMMIT DROP */ +} SVariableXActAction; + +typedef struct SVariableXActActionItem +{ + Oid varid; /* varid of session variable */ + + /* + * creating_subid is the ID of the creating subxact. If the action was + * unregistered during the current transaction, deleting_subid is the ID + * of the deleting subxact, otherwise InvalidSubTransactionId. + */ + SubTransactionId creating_subid; + SubTransactionId deleting_subid; +} SVariableXActActionItem; + +/* List holds fields of SVariableXActActionItem type */ +static List *xact_drop_actions = NIL; + +static void register_session_variable_xact_action(Oid varid, SVariableXActAction action); +static void unregister_session_variable_xact_action(Oid varid, SVariableXActAction action); + + +/* + * Do the necessary work to setup local memory management of a new + * variable. + * + * Caller should already have created the necessary entry in catalog + * and made them visible. + */ +void +SessionVariableCreatePostprocess(Oid varid, char eoxaction) +{ + /* + * For temporary variables, we need to create a new end of xact action to + * ensure deletion from catalog. + */ + if (eoxaction == VARIABLE_EOX_DROP) + { + Assert(isTempNamespace(get_session_variable_namespace(varid))); + + register_session_variable_xact_action(varid, SVAR_ON_COMMIT_DROP); + } +} + +/* + * Handle the local memory cleanup for a DROP VARIABLE command. + * + * Caller should take care of removing the pg_variable entry first. + */ +void +SessionVariableDropPostprocess(Oid varid) +{ + /* + * The entry was removed from catalog already, we must not do it + * again at end of xact time. + */ + unregister_session_variable_xact_action(varid, SVAR_ON_COMMIT_DROP); +} + +/* + * Registration of actions to be executed on session variables at transaction + * end time. We want to drop temporary session variables with clause ON COMMIT + * DROP, or we want to reset values of session variables with clause ON + * TRANSACTION END RESET or we want to clean (reset) local memory allocated by + * values of dropped session variables. + */ + +/* + * Register a session variable xact action. + */ +static void +register_session_variable_xact_action(Oid varid, + SVariableXActAction action) +{ + SVariableXActActionItem *xact_ai; + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(CacheMemoryContext); + + xact_ai = (SVariableXActActionItem *) + palloc(sizeof(SVariableXActActionItem)); + + xact_ai->varid = varid; + + xact_ai->creating_subid = GetCurrentSubTransactionId(); + xact_ai->deleting_subid = InvalidSubTransactionId; + + Assert(action == SVAR_ON_COMMIT_DROP); + xact_drop_actions = lcons(xact_ai, xact_drop_actions); + + MemoryContextSwitchTo(oldcxt); +} + +/* + * Unregister an action on a given session variable from action list. In this + * moment, the action is just marked as deleted by setting deleting_subid. The + * calling even might be rollbacked, in which case we should not lose this + * action. + */ +static void +unregister_session_variable_xact_action(Oid varid, + SVariableXActAction action) +{ + ListCell *l; + + Assert(action == SVAR_ON_COMMIT_DROP); + + foreach(l, xact_drop_actions) + { + SVariableXActActionItem *xact_ai = + (SVariableXActActionItem *) lfirst(l); + + if (xact_ai->varid == varid) + xact_ai->deleting_subid = GetCurrentSubTransactionId(); + } +} + +/* + * Perform ON TRANSACTION END RESET or ON COMMIT DROP + * and COMMIT/ROLLBACK of transaction session variables. + */ +void +AtPreEOXact_SessionVariable(bool isCommit) +{ + ListCell *l; + + foreach(l, xact_drop_actions) + { + SVariableXActActionItem *xact_ai = + (SVariableXActActionItem *) lfirst(l); + + /* Iterate only over entries that are still pending */ + if (xact_ai->deleting_subid == InvalidSubTransactionId) + { + + /* + * ON COMMIT DROP is allowed only for temp session variables. So + * we should explicitly delete only when current transaction was + * committed. When it's rollback, then session variable is removed + * automatically. + */ + if (isCommit) + { + ObjectAddress object; + + object.classId = VariableRelationId; + object.objectId = xact_ai->varid; + object.objectSubId = 0; + + /* + * Since this is an automatic drop, rather than one directly + * initiated by the user, we pass the + * PERFORM_DELETION_INTERNAL flag. + */ + elog(DEBUG1, "session variable (oid:%u) will be deleted (forced by SVAR_ON_COMMIT_DROP action)", + xact_ai->varid); + + performDeletion(&object, DROP_CASCADE, + PERFORM_DELETION_INTERNAL | + PERFORM_DELETION_QUIETLY); + } + } + } + + /* + * Any drop action left is an entry that was unregistered and not + * rollbacked, so we can simply remove them. + */ + list_free_deep(xact_drop_actions); + xact_drop_actions = NIL; +} + +/* + * Post-subcommit or post-subabort cleanup of xact action list. + * + * During subabort, we can immediately remove entries created during this + * subtransaction. During subcommit, just transfer entries marked during + * this subtransaction as being the parent's responsibility. + */ +void +AtEOSubXact_SessionVariable(bool isCommit, + SubTransactionId mySubid, + SubTransactionId parentSubid) +{ + ListCell *cur_item; + + foreach(cur_item, xact_drop_actions) + { + SVariableXActActionItem *xact_ai = + (SVariableXActActionItem *) lfirst(cur_item); + + if (!isCommit && xact_ai->creating_subid == mySubid) + { + /* cur_item must be removed */ + xact_drop_actions = foreach_delete_current(xact_drop_actions, cur_item); + pfree(xact_ai); + } + else + { + /* cur_item must be preserved */ + if (xact_ai->creating_subid == mySubid) + xact_ai->creating_subid = parentSubid; + if (xact_ai->deleting_subid == mySubid) + xact_ai->deleting_subid = isCommit ? parentSubid : InvalidSubTransactionId; + } + } +} diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index c7a8a689b7..c99f091f1f 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -47,6 +47,7 @@ #include "catalog/pg_tablespace.h" #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "catalog/storage.h" #include "catalog/storage_xlog.h" #include "catalog/toasting.h" @@ -6487,6 +6488,8 @@ ATTypedTableRecursion(List **wqueue, Relation rel, AlterTableCmd *cmd, * Eventually, we'd like to propagate the check or rewrite operation * into such tables, but for now, just error out if we find any. * + * Check if the type "typeOid" is used as type of some session variable too. + * * Caller should provide either the associated relation of a rowtype, * or a type name (not both) for use in the error message, if any. * @@ -6549,6 +6552,45 @@ find_composite_type_dependencies(Oid typeOid, Relation origRelation, continue; } + /* Don't allow change of type used by session's variable */ + if (pg_depend->classid == VariableRelationId) + { + Oid varid = pg_depend->objid; + + if (origTypeName) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot alter type \"%s\" because session variable \"%s.%s\" uses it", + origTypeName, + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)))); + else if (origRelation->rd_rel->relkind == RELKIND_COMPOSITE_TYPE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot alter type \"%s\" because session variable \"%s.%s\" uses it", + RelationGetRelationName(origRelation), + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)))); + else if (origRelation->rd_rel->relkind == RELKIND_FOREIGN_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot alter foreign table \"%s\" because session variable \"%s.%s\" uses it", + RelationGetRelationName(origRelation), + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)))); + else if (origRelation->rd_rel->relkind == RELKIND_RELATION || + origRelation->rd_rel->relkind == RELKIND_MATVIEW || + origRelation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot alter table \"%s\" because session variable \"%s.%s\" uses it", + RelationGetRelationName(origRelation), + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)))); + + continue; + } + /* Else, ignore dependees that aren't relations */ if (pg_depend->classid != RelationRelationId) continue; @@ -12835,6 +12877,7 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, case OCLASS_PUBLICATION_REL: case OCLASS_SUBSCRIPTION: case OCLASS_TRANSFORM: + case OCLASS_VARIABLE: /* * We don't expect any of these sorts of objects to depend on diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index d6426f3b8e..c9a93a3c7a 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -53,6 +53,7 @@ #include "catalog/namespace.h" #include "catalog/pg_am.h" #include "catalog/pg_trigger.h" +#include "catalog/pg_variable.h" #include "commands/defrem.h" #include "commands/trigger.h" #include "gramparse.h" @@ -292,8 +293,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt - CreateSchemaStmt CreateSeqStmt CreateStmt CreateStatsStmt CreateTableSpaceStmt - CreateFdwStmt CreateForeignServerStmt CreateForeignTableStmt + CreateSchemaStmt CreateSessionVarStmt CreateSeqStmt CreateStmt CreateStatsStmt + CreateTableSpaceStmt CreateFdwStmt CreateForeignServerStmt CreateForeignTableStmt CreateAssertionStmt CreateTransformStmt CreateTrigStmt CreateEventTrigStmt CreateUserStmt CreateUserMappingStmt CreateRoleStmt CreatePolicyStmt CreatedbStmt DeclareCursorStmt DefineStmt DeleteStmt DiscardStmt DoStmt @@ -473,6 +474,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <ival> OptTemp %type <ival> OptNoLog %type <oncommit> OnCommitOption +%type <ival> OnEOXActionOption %type <ival> for_locking_strength %type <node> for_locking_item @@ -642,6 +644,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <partboundspec> PartitionBoundSpec %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <node> OptSessionVarDefExpr +%type <boolean> OptNotNull OptImmutable %type <node> json_format_clause_opt @@ -770,8 +774,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); UESCAPE UNBOUNDED UNCOMMITTED UNENCRYPTED UNION UNIQUE UNKNOWN UNLISTEN UNLOGGED UNTIL UPDATE USER USING - VACUUM VALID VALIDATE VALIDATOR VALUE_P VALUES VARCHAR VARIADIC VARYING - VERBOSE VERSION_P VIEW VIEWS VOLATILE + VACUUM VALID VALIDATE VALIDATOR VALUE_P VALUES VARCHAR VARIABLE VARIABLES + VARIADIC VARYING VERBOSE VERSION_P VIEW VIEWS VOLATILE WHEN WHERE WHITESPACE_P WINDOW WITH WITHIN WITHOUT WORK WRAPPER WRITE @@ -1025,6 +1029,7 @@ stmt: | CreatePolicyStmt | CreatePLangStmt | CreateSchemaStmt + | CreateSessionVarStmt | CreateSeqStmt | CreateStmt | CreateSubscriptionStmt @@ -1569,6 +1574,7 @@ schema_stmt: | CreateTrigStmt | GrantStmt | ViewStmt + | CreateSessionVarStmt ; @@ -5054,6 +5060,69 @@ create_extension_opt_item: } ; +/***************************************************************************** + * + * QUERY : + * CREATE VARIABLE varname [AS] type + * + *****************************************************************************/ + +CreateSessionVarStmt: + CREATE OptTemp OptImmutable VARIABLE qualified_name opt_as Typename opt_collate_clause OptNotNull OptSessionVarDefExpr OnEOXActionOption + { + CreateSessionVarStmt *n = makeNode(CreateSessionVarStmt); + $5->relpersistence = $2; + n->is_immutable = $3; + n->variable = $5; + n->typeName = $7; + n->collClause = (CollateClause *) $8; + n->is_not_null = $9; + n->defexpr = $10; + n->eoxaction = $11; + n->if_not_exists = false; + $$ = (Node *) n; + } + | CREATE OptTemp OptImmutable VARIABLE IF_P NOT EXISTS qualified_name opt_as Typename opt_collate_clause OptNotNull OptSessionVarDefExpr OnEOXActionOption + { + CreateSessionVarStmt *n = makeNode(CreateSessionVarStmt); + $8->relpersistence = $2; + n->is_immutable = $3; + n->variable = $8; + n->typeName = $10; + n->collClause = (CollateClause *) $11; + n->is_not_null = $12; + n->defexpr = $13; + n->eoxaction = $14; + n->if_not_exists = true; + $$ = (Node *) n; + } + ; + +OptSessionVarDefExpr: DEFAULT b_expr { $$ = $2; } + | /* EMPTY */ { $$ = NULL; } + ; + +/* + * Temporary session variables can be dropped on successful + * transaction end like tables. RESET can only be forced on + * transaction end. Since the session variables are not + * transactional, we have to handle ROLLBACK too. + * The clause ON TRANSACTION END is clearer than some + * ON COMMIT ROLLBACK RESET clause. + */ +OnEOXActionOption: ON COMMIT DROP { $$ = VARIABLE_EOX_DROP; } + | ON TRANSACTION END_P RESET { $$ = VARIABLE_EOX_RESET; } + | /*EMPTY*/ { $$ = VARIABLE_EOX_NOOP; } + ; + +OptNotNull: NOT NULL_P { $$ = true; } + | /* EMPTY */ { $$ = false; } + ; + +OptImmutable: IMMUTABLE { $$ = true; } + | /* EMPTY */ { $$ = false; } + ; + /***************************************************************************** * * ALTER EXTENSION name UPDATE [ TO version ] @@ -6831,6 +6900,7 @@ object_type_any_name: | TEXT_P SEARCH DICTIONARY { $$ = OBJECT_TSDICTIONARY; } | TEXT_P SEARCH TEMPLATE { $$ = OBJECT_TSTEMPLATE; } | TEXT_P SEARCH CONFIGURATION { $$ = OBJECT_TSCONFIGURATION; } + | VARIABLE { $$ = OBJECT_VARIABLE; } ; /* @@ -7707,6 +7777,14 @@ privilege_target: n->objs = $2; $$ = n; } + | VARIABLE qualified_name_list + { + PrivTarget *n = (PrivTarget *) palloc(sizeof(PrivTarget)); + n->targtype = ACL_TARGET_OBJECT; + n->objtype = OBJECT_VARIABLE; + n->objs = $2; + $$ = n; + } | ALL TABLES IN_P SCHEMA name_list { PrivTarget *n = (PrivTarget *) palloc(sizeof(PrivTarget)); @@ -7752,6 +7830,14 @@ privilege_target: n->objs = $5; $$ = n; } + | ALL VARIABLES IN_P SCHEMA name_list + { + PrivTarget *n = (PrivTarget *) palloc(sizeof(PrivTarget)); + n->targtype = ACL_TARGET_ALL_IN_SCHEMA; + n->objtype = OBJECT_VARIABLE; + n->objs = $5; + $$ = n; + } ; @@ -7949,6 +8035,7 @@ defacl_privilege_target: | SEQUENCES { $$ = OBJECT_SEQUENCE; } | TYPES_P { $$ = OBJECT_TYPE; } | SCHEMAS { $$ = OBJECT_SCHEMA; } + | VARIABLES { $$ = OBJECT_VARIABLE; } ; @@ -9731,6 +9818,25 @@ RenameStmt: ALTER AGGREGATE aggregate_with_argtypes RENAME TO name n->missing_ok = false; $$ = (Node *) n; } + | ALTER VARIABLE any_name RENAME TO name + { + RenameStmt *n = makeNode(RenameStmt); + n->renameType = OBJECT_VARIABLE; + n->object = (Node *) $3; + n->newname = $6; + n->missing_ok = false; + $$ = (Node *)n; + } + | ALTER VARIABLE IF_P EXISTS any_name RENAME TO name + { + RenameStmt *n = makeNode(RenameStmt); + n->renameType = OBJECT_VARIABLE; + n->object = (Node *) $5; + n->newname = $8; + n->missing_ok = true; + $$ = (Node *)n; + } + ; opt_column: COLUMN @@ -10092,6 +10198,25 @@ AlterObjectSchemaStmt: n->missing_ok = false; $$ = (Node *) n; } + | ALTER VARIABLE any_name SET SCHEMA name + { + AlterObjectSchemaStmt *n = makeNode(AlterObjectSchemaStmt); + n->objectType = OBJECT_VARIABLE; + n->object = (Node *) $3; + n->newschema = $6; + n->missing_ok = false; + $$ = (Node *)n; + } + | ALTER VARIABLE IF_P EXISTS any_name SET SCHEMA name + { + AlterObjectSchemaStmt *n = makeNode(AlterObjectSchemaStmt); + n->objectType = OBJECT_VARIABLE; + n->object = (Node *) $5; + n->newschema = $8; + n->missing_ok = true; + $$ = (Node *)n; + } + ; /***************************************************************************** @@ -10371,6 +10496,14 @@ AlterOwnerStmt: ALTER AGGREGATE aggregate_with_argtypes OWNER TO RoleSpec n->newowner = $6; $$ = (Node *) n; } + | ALTER VARIABLE any_name OWNER TO RoleSpec + { + AlterOwnerStmt *n = makeNode(AlterOwnerStmt); + n->objectType = OBJECT_VARIABLE; + n->object = (Node *) $3; + n->newowner = $6; + $$ = (Node *)n; + } ; @@ -17299,6 +17432,8 @@ unreserved_keyword: | VALIDATE | VALIDATOR | VALUE_P + | VARIABLE + | VARIABLES | VARYING | VERSION_P | VIEW @@ -17924,6 +18059,8 @@ bare_label_keyword: | VALUE_P | VALUES | VARCHAR + | VARIABLE + | VARIABLES | VARIADIC | VERBOSE | VERSION_P diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..6fc03166c6 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -472,6 +472,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_COLUMN_DEFAULT: case EXPR_KIND_FUNCTION_DEFAULT: + case EXPR_KIND_VARIABLE_DEFAULT: if (isAgg) err = _("aggregate functions are not allowed in DEFAULT expressions"); @@ -915,6 +916,7 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, break; case EXPR_KIND_COLUMN_DEFAULT: case EXPR_KIND_FUNCTION_DEFAULT: + case EXPR_KIND_VARIABLE_DEFAULT: err = _("window functions are not allowed in DEFAULT expressions"); break; case EXPR_KIND_INDEX_EXPRESSION: diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64356436ef..c69259c9e3 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -18,6 +18,7 @@ #include "catalog/pg_aggregate.h" #include "catalog/pg_proc.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "commands/dbcommands.h" #include "miscadmin.h" #include "nodes/makefuncs.h" @@ -534,6 +535,8 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_VARIABLE_DEFAULT: + /* okay */ break; @@ -1755,6 +1758,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) break; case EXPR_KIND_COLUMN_DEFAULT: case EXPR_KIND_FUNCTION_DEFAULT: + case EXPR_KIND_VARIABLE_DEFAULT: err = _("cannot use subquery in DEFAULT expression"); break; case EXPR_KIND_INDEX_EXPRESSION: @@ -3048,6 +3052,7 @@ ParseExprKindName(ParseExprKind exprKind) return "CHECK"; case EXPR_KIND_COLUMN_DEFAULT: case EXPR_KIND_FUNCTION_DEFAULT: + case EXPR_KIND_VARIABLE_DEFAULT: return "DEFAULT"; case EXPR_KIND_INDEX_EXPRESSION: return "index expression"; diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..a0fdaa8605 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2618,6 +2618,7 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) break; case EXPR_KIND_COLUMN_DEFAULT: case EXPR_KIND_FUNCTION_DEFAULT: + case EXPR_KIND_VARIABLE_DEFAULT: err = _("set-returning functions are not allowed in DEFAULT expressions"); break; case EXPR_KIND_INDEX_EXPRESSION: diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b1255e3b70..e7e75209dd 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -105,6 +105,7 @@ typedef struct List *indexes; /* CREATE INDEX items */ List *triggers; /* CREATE TRIGGER items */ List *grants; /* GRANT items */ + List *variables; /* CREATE VARIABLE items */ } CreateSchemaStmtContext; @@ -3822,6 +3823,7 @@ transformCreateSchemaStmtElements(List *schemaElts, const char *schemaName) cxt.indexes = NIL; cxt.triggers = NIL; cxt.grants = NIL; + cxt.variables = NIL; /* * Run through each schema element in the schema element list. Separate @@ -3890,6 +3892,15 @@ transformCreateSchemaStmtElements(List *schemaElts, const char *schemaName) cxt.grants = lappend(cxt.grants, element); break; + case T_CreateSessionVarStmt: + { + CreateSessionVarStmt *elp = (CreateSessionVarStmt *) element; + + setSchemaName(cxt.schemaname, &elp->variable->schemaname); + cxt.variables = lappend(cxt.variables, element); + } + break; + default: elog(ERROR, "unrecognized node type: %d", (int) nodeTag(element)); @@ -3903,6 +3914,7 @@ transformCreateSchemaStmtElements(List *schemaElts, const char *schemaName) result = list_concat(result, cxt.indexes); result = list_concat(result, cxt.triggers); result = list_concat(result, cxt.grants); + result = list_concat(result, cxt.variables); return result; } diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 30b51bf4d3..ddc26794b2 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -49,6 +49,7 @@ #include "commands/proclang.h" #include "commands/publicationcmds.h" #include "commands/schemacmds.h" +#include "commands/session_variable.h" #include "commands/seclabel.h" #include "commands/sequence.h" #include "commands/subscriptioncmds.h" @@ -189,6 +190,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree) case T_CreateRangeStmt: case T_CreateRoleStmt: case T_CreateSchemaStmt: + case T_CreateSessionVarStmt: case T_CreateSeqStmt: case T_CreateStatsStmt: case T_CreateStmt: @@ -1396,6 +1398,10 @@ ProcessUtilitySlow(ParseState *pstate, } break; + case T_CreateSessionVarStmt: + address = CreateVariable(pstate, (CreateSessionVarStmt *) parsetree); + break; + /* * ************* object creation / destruction ************** */ @@ -2342,6 +2348,9 @@ AlterObjectTypeCommandTag(ObjectType objtype) case OBJECT_STATISTIC_EXT: tag = CMDTAG_ALTER_STATISTICS; break; + case OBJECT_VARIABLE: + tag = CMDTAG_ALTER_VARIABLE; + break; default: tag = CMDTAG_UNKNOWN; break; @@ -2650,6 +2659,9 @@ CreateCommandTag(Node *parsetree) case OBJECT_STATISTIC_EXT: tag = CMDTAG_DROP_STATISTICS; break; + case OBJECT_VARIABLE: + tag = CMDTAG_DROP_VARIABLE; + break; default: tag = CMDTAG_UNKNOWN; } @@ -3226,6 +3238,10 @@ CreateCommandTag(Node *parsetree) } break; + case T_CreateSessionVarStmt: + tag = CMDTAG_CREATE_VARIABLE; + break; + default: elog(WARNING, "unrecognized node type: %d", (int) nodeTag(parsetree)); diff --git a/src/backend/utils/adt/acl.c b/src/backend/utils/adt/acl.c index c660fd3e70..dc09de12b9 100644 --- a/src/backend/utils/adt/acl.c +++ b/src/backend/utils/adt/acl.c @@ -844,6 +844,10 @@ acldefault(ObjectType objtype, Oid ownerId) world_default = ACL_NO_RIGHTS; owner_default = ACL_ALL_RIGHTS_PARAMETER_ACL; break; + case OBJECT_VARIABLE: + world_default = ACL_NO_RIGHTS; + owner_default = ACL_ALL_RIGHTS_VARIABLE; + break; default: elog(ERROR, "unrecognized object type: %d", (int) objtype); world_default = ACL_NO_RIGHTS; /* keep compiler quiet */ @@ -941,6 +945,9 @@ acldefault_sql(PG_FUNCTION_ARGS) case 'T': objtype = OBJECT_TYPE; break; + case 'V': + objtype = OBJECT_VARIABLE; + break; default: elog(ERROR, "unrecognized object type abbreviation: %c", objtypec); } diff --git a/src/backend/utils/cache/lsyscache.c b/src/backend/utils/cache/lsyscache.c index c7607895cd..703e261de7 100644 --- a/src/backend/utils/cache/lsyscache.c +++ b/src/backend/utils/cache/lsyscache.c @@ -36,6 +36,7 @@ #include "catalog/pg_subscription.h" #include "catalog/pg_transform.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "miscadmin.h" #include "nodes/makefuncs.h" #include "utils/array.h" @@ -3672,3 +3673,115 @@ get_subscription_name(Oid subid, bool missing_ok) return subname; } + +/* ---------- PG_VARIABLE CACHE ---------- */ + +/* + * get_varname_varid + * Given name and namespace of variable, look up the OID. + */ +Oid +get_varname_varid(const char *varname, Oid varnamespace) +{ + return GetSysCacheOid2(VARIABLENAMENSP, Anum_pg_variable_oid, + PointerGetDatum(varname), + ObjectIdGetDatum(varnamespace)); +} + +/* + * get_session_variable_name + * Returns a palloc'd copy of the name of a given session variable. + */ +char * +get_session_variable_name(Oid varid) +{ + HeapTuple tup; + Form_pg_variable varform; + char *varname; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for session variable %u", varid); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + varname = pstrdup(NameStr(varform->varname)); + + ReleaseSysCache(tup); + + return varname; +} + +/* + * get_session_variable_namespace + * Returns the pg_namespace OID associated with a given session variable. + */ +Oid +get_session_variable_namespace(Oid varid) +{ + HeapTuple tup; + Form_pg_variable varform; + Oid varnamespace; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for variable %u", varid); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + varnamespace = varform->varnamespace; + + ReleaseSysCache(tup); + + return varnamespace; +} + +/* + * Returns the type, typmod and collid of the given session variable. + */ +Oid +get_session_variable_type(Oid varid) +{ + HeapTuple tup; + Form_pg_variable varform; + Oid vartype; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for session variable %u", varid); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + vartype = varform->vartype; + + ReleaseSysCache(tup); + + return vartype; +} + +/* + * Returns the type, typmod and collid of the given session variable. + */ +void +get_session_variable_type_typmod_collid(Oid varid, Oid *typid, int32 *typmod, + Oid *collid) +{ + HeapTuple tup; + Form_pg_variable varform; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for session variable %u", varid); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + *typid = varform->vartype; + *typmod = varform->vartypmod; + *collid = varform->varcollation; + + ReleaseSysCache(tup); +} diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c index 4e4a34bde8..d0e5265519 100644 --- a/src/backend/utils/cache/syscache.c +++ b/src/backend/utils/cache/syscache.c @@ -75,6 +75,7 @@ #include "catalog/pg_ts_template.h" #include "catalog/pg_type.h" #include "catalog/pg_user_mapping.h" +#include "catalog/pg_variable.h" #include "lib/qunique.h" #include "utils/catcache.h" #include "utils/lsyscache.h" @@ -676,6 +677,19 @@ static const struct cachedesc cacheinfo[] = { KEY(Anum_pg_user_mapping_umuser, Anum_pg_user_mapping_umserver), 2 + }, + [VARIABLENAMENSP] = { + VariableRelationId, + VariableNameNspIndexId, + KEY(Anum_pg_variable_varname, + Anum_pg_variable_varnamespace), + 8 + }, + [VARIABLEOID] = { + VariableRelationId, + VariableObjectIndexId, + KEY(Anum_pg_variable_oid), + 8 } }; diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h index ffd5e9dc82..e43a1ab96a 100644 --- a/src/include/catalog/dependency.h +++ b/src/include/catalog/dependency.h @@ -127,10 +127,11 @@ typedef enum ObjectClass OCLASS_PUBLICATION_NAMESPACE, /* pg_publication_namespace */ OCLASS_PUBLICATION_REL, /* pg_publication_rel */ OCLASS_SUBSCRIPTION, /* pg_subscription */ - OCLASS_TRANSFORM /* pg_transform */ + OCLASS_TRANSFORM, /* pg_transform */ + OCLASS_VARIABLE /* pg_variable */ } ObjectClass; -#define LAST_OCLASS OCLASS_TRANSFORM +#define LAST_OCLASS OCLASS_VARIABLE /* flag bits for performDeletion/performMultipleDeletions: */ #define PERFORM_DELETION_INTERNAL 0x0001 /* internal action */ diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build index 3179be09d3..51414b93bd 100644 --- a/src/include/catalog/meson.build +++ b/src/include/catalog/meson.build @@ -65,6 +65,7 @@ catalog_headers = [ 'pg_publication_rel.h', 'pg_subscription.h', 'pg_subscription_rel.h', + 'pg_variable.h', ] bki_data = [ diff --git a/src/include/catalog/namespace.h b/src/include/catalog/namespace.h index f64a0ec26b..787de15ed1 100644 --- a/src/include/catalog/namespace.h +++ b/src/include/catalog/namespace.h @@ -96,6 +96,8 @@ extern Oid TypenameGetTypid(const char *typname); extern Oid TypenameGetTypidExtended(const char *typname, bool temp_ok); extern bool TypeIsVisible(Oid typid); +extern bool VariableIsVisible(Oid varid); + extern FuncCandidateList FuncnameGetCandidates(List *names, int nargs, List *argnames, bool expand_variadic, @@ -164,6 +166,9 @@ extern void SetTempNamespaceState(Oid tempNamespaceId, Oid tempToastNamespaceId); extern void ResetTempTableNamespace(void); +extern List *NamesFromList(List *names); +extern Oid LookupVariable(const char *nspname, const char *varname, bool missing_ok); + extern OverrideSearchPath *GetOverrideSearchPath(MemoryContext context); extern OverrideSearchPath *CopyOverrideSearchPath(OverrideSearchPath *path); extern bool OverrideSearchPathMatchesCurrent(OverrideSearchPath *path); diff --git a/src/include/catalog/pg_default_acl.h b/src/include/catalog/pg_default_acl.h index d6d0a03f0c..53affc969c 100644 --- a/src/include/catalog/pg_default_acl.h +++ b/src/include/catalog/pg_default_acl.h @@ -66,6 +66,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_default_acl_oid_index, 828, DefaultAclOidIndexId, o #define DEFACLOBJ_FUNCTION 'f' /* function */ #define DEFACLOBJ_TYPE 'T' /* type */ #define DEFACLOBJ_NAMESPACE 'n' /* namespace */ +#define DEFACLOBJ_VARIABLE 'V' /* variable */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index b2bc81b15f..258a2ed903 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6399,6 +6399,9 @@ proname => 'pg_collation_is_visible', procost => '10', provolatile => 's', prorettype => 'bool', proargtypes => 'oid', prosrc => 'pg_collation_is_visible' }, +{ oid => '9221', descr => 'is session variable visible in search path?', + proname => 'pg_variable_is_visible', procost => '10', provolatile => 's', + prorettype => 'bool', proargtypes => 'oid', prosrc => 'pg_variable_is_visible' }, { oid => '2854', descr => 'get OID of current session\'s temp schema, if any', proname => 'pg_my_temp_schema', provolatile => 's', proparallel => 'r', diff --git a/src/include/catalog/pg_variable.h b/src/include/catalog/pg_variable.h new file mode 100644 index 0000000000..698cd73c7e --- /dev/null +++ b/src/include/catalog/pg_variable.h @@ -0,0 +1,130 @@ +/*------------------------------------------------------------------------- + * + * pg_variable.h + * definition of session variables system catalog (pg_variables) + * + * + * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/catalog/pg_variable.h + * + * NOTES + * The Catalog.pm module reads this file and derives schema + * information. + * + *------------------------------------------------------------------------- + */ +#ifndef PG_VARIABLE_H +#define PG_VARIABLE_H + +#include "catalog/genbki.h" +#include "catalog/objectaddress.h" +#include "catalog/pg_variable_d.h" +#include "utils/acl.h" + +/* ---------------- + * pg_variable definition. cpp turns this into + * typedef struct FormData_pg_variable + * + * The column varcreate_lsn of XlogRecPtr type (8-byte) should be on position + * divisible by 8 unconditionally and before varname column of NameData type. + * see sanity_check:check_columns + * ---------------- + */ +CATALOG(pg_variable,9222,VariableRelationId) +{ + Oid oid; /* oid */ + + /* OID of entry in pg_type for variable's type */ + Oid vartype BKI_LOOKUP(pg_type); + + /* Used for identity check [oid, create_lsn] */ + XLogRecPtr varcreate_lsn; + + /* variable name */ + NameData varname; + + /* OID of namespace containing variable class */ + Oid varnamespace BKI_LOOKUP(pg_namespace); + + /* variable owner */ + Oid varowner BKI_LOOKUP(pg_authid); + + /* typmod for variable's type */ + int32 vartypmod; + + /* Don't allow NULL */ + bool varisnotnull; + + /* Don't allow changes */ + bool varisimmutable; + + /* action on transaction end */ + char vareoxaction; + + /* variable collation */ + Oid varcollation BKI_LOOKUP_OPT(pg_collation); + + +#ifdef CATALOG_VARLEN /* variable-length fields start here */ + + /* access permissions */ + aclitem varacl[1] BKI_DEFAULT(_null_); + + /* list of expression trees for variable default (NULL if none) */ + pg_node_tree vardefexpr BKI_DEFAULT(_null_); + +#endif +} FormData_pg_variable; + +typedef enum VariableEOXAction +{ + VARIABLE_EOX_NOOP = 'n', /* NOOP */ + VARIABLE_EOX_DROP = 'd', /* ON COMMIT DROP */ + VARIABLE_EOX_RESET = 'r', /* ON TRANSACTION END RESET */ +} VariableEOXAction; + +/* ---------------- + * Form_pg_variable corresponds to a pointer to a tuple with + * the format of pg_variable relation. + * ---------------- + */ +typedef FormData_pg_variable *Form_pg_variable; + +DECLARE_TOAST(pg_variable, 9223, 9224); + +DECLARE_UNIQUE_INDEX_PKEY(pg_variable_oid_index, 9225, VariableOidIndexId, on pg_variable using btree(oid oid_ops)); +#define VariableObjectIndexId 9225 + +DECLARE_UNIQUE_INDEX(pg_variable_varname_nsp_index, 9226, VariableNameNspIndexId, on pg_variable using btree(varname name_ops, varnamespace oid_ops)); +#define VariableNameNspIndexId 9226 + +/* + * The Variable struct is based on FormData_pg_variable struct. Unlike + * FormData_pg_variable it can hold node of deserialized expression used + * for calculation of default value. + */ +typedef struct Variable +{ + Oid oid; + Oid typid; + XLogRecPtr create_lsn; + char *name; + Oid namespaceid; + Oid owner; + int32 typmod; + bool is_not_null; + bool is_immutable; + VariableEOXAction eoxaction; + Oid collation; + bool has_defexpr; + Node *defexpr; +} Variable; + +extern ObjectAddress CreateVariable(ParseState *pstate, + CreateSessionVarStmt *stmt); +extern void DropVariable(Oid varid); +extern void InitVariable(Variable *var, Oid varid, bool fast_only); + +#endif /* PG_VARIABLE_H */ diff --git a/src/include/commands/session_variable.h b/src/include/commands/session_variable.h new file mode 100644 index 0000000000..343ee070a5 --- /dev/null +++ b/src/include/commands/session_variable.h @@ -0,0 +1,34 @@ +/*------------------------------------------------------------------------- + * + * sessionvariable.h + * prototypes for sessionvariable.c. + * + * + * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/commands/session_variable.h + * + *------------------------------------------------------------------------- + */ + +#ifndef SESSIONVARIABLE_H +#define SESSIONVARIABLE_H + +#include "catalog/objectaddress.h" +#include "catalog/pg_variable.h" +#include "nodes/params.h" +#include "nodes/parsenodes.h" +#include "nodes/plannodes.h" +#include "tcop/cmdtag.h" +#include "utils/queryenvironment.h" + +extern void SessionVariableCreatePostprocess(Oid varid, char eoxaction); +extern void SessionVariableDropPostprocess(Oid varid); + +extern void AtPreEOXact_SessionVariable(bool isCommit); +extern void AtEOSubXact_SessionVariable(bool isCommit, + SubTransactionId mySubid, + SubTransactionId parentSubid); + +#endif diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index cc7b32b279..eecbbd7768 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -2130,6 +2130,7 @@ typedef enum ObjectType OBJECT_TSTEMPLATE, OBJECT_TYPE, OBJECT_USER_MAPPING, + OBJECT_VARIABLE, OBJECT_VIEW } ObjectType; @@ -3253,6 +3254,25 @@ typedef struct AlterStatsStmt bool missing_ok; /* skip error if statistics object is missing */ } AlterStatsStmt; + +/* ---------------------- + * {Create|Alter} VARIABLE Statement + * ---------------------- + */ +typedef struct CreateSessionVarStmt +{ + NodeTag type; + RangeVar *variable; /* the variable to create */ + TypeName *typeName; /* the type of variable */ + CollateClause *collClause; + Node *defexpr; /* default expression */ + char eoxaction; /* on commit action */ + bool if_not_exists; /* do nothing if it already exists */ + bool is_not_null; /* Disallow nulls */ + bool is_immutable; /* Don't allow changes */ +} CreateSessionVarStmt; + + /* ---------------------- * Create Function Statement * ---------------------- diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index f5b2e61ca5..9f79b90251 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -464,6 +464,8 @@ PG_KEYWORD("validator", VALIDATOR, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("value", VALUE_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("values", VALUES, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("varchar", VARCHAR, COL_NAME_KEYWORD, BARE_LABEL) +PG_KEYWORD("variable", VARIABLE, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("variables", VARIABLES, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("variadic", VARIADIC, RESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("varying", VARYING, UNRESERVED_KEYWORD, AS_LABEL) PG_KEYWORD("verbose", VERBOSE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h index f589112d5e..eccde83628 100644 --- a/src/include/parser/parse_node.h +++ b/src/include/parser/parse_node.h @@ -81,6 +81,7 @@ typedef enum ParseExprKind EXPR_KIND_COPY_WHERE, /* WHERE condition in COPY FROM */ EXPR_KIND_GENERATED_COLUMN, /* generation expression for a column */ EXPR_KIND_CYCLE_MARK, /* cycle mark value */ + EXPR_KIND_VARIABLE_DEFAULT, /* default value for session variable */ } ParseExprKind; diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h index e738ac1c09..259bdc994e 100644 --- a/src/include/tcop/cmdtaglist.h +++ b/src/include/tcop/cmdtaglist.h @@ -68,6 +68,7 @@ PG_CMDTAG(CMDTAG_ALTER_TRANSFORM, "ALTER TRANSFORM", true, false, false) PG_CMDTAG(CMDTAG_ALTER_TRIGGER, "ALTER TRIGGER", true, false, false) PG_CMDTAG(CMDTAG_ALTER_TYPE, "ALTER TYPE", true, true, false) PG_CMDTAG(CMDTAG_ALTER_USER_MAPPING, "ALTER USER MAPPING", true, false, false) +PG_CMDTAG(CMDTAG_ALTER_VARIABLE, "ALTER VARIABLE", true, false, false) PG_CMDTAG(CMDTAG_ALTER_VIEW, "ALTER VIEW", true, false, false) PG_CMDTAG(CMDTAG_ANALYZE, "ANALYZE", false, false, false) PG_CMDTAG(CMDTAG_BEGIN, "BEGIN", false, false, false) @@ -123,6 +124,7 @@ PG_CMDTAG(CMDTAG_CREATE_TRANSFORM, "CREATE TRANSFORM", true, false, false) PG_CMDTAG(CMDTAG_CREATE_TRIGGER, "CREATE TRIGGER", true, false, false) PG_CMDTAG(CMDTAG_CREATE_TYPE, "CREATE TYPE", true, false, false) PG_CMDTAG(CMDTAG_CREATE_USER_MAPPING, "CREATE USER MAPPING", true, false, false) +PG_CMDTAG(CMDTAG_CREATE_VARIABLE, "CREATE VARIABLE", true, false, false) PG_CMDTAG(CMDTAG_CREATE_VIEW, "CREATE VIEW", true, false, false) PG_CMDTAG(CMDTAG_DEALLOCATE, "DEALLOCATE", false, false, false) PG_CMDTAG(CMDTAG_DEALLOCATE_ALL, "DEALLOCATE ALL", false, false, false) @@ -175,6 +177,7 @@ PG_CMDTAG(CMDTAG_DROP_TRANSFORM, "DROP TRANSFORM", true, false, false) PG_CMDTAG(CMDTAG_DROP_TRIGGER, "DROP TRIGGER", true, false, false) PG_CMDTAG(CMDTAG_DROP_TYPE, "DROP TYPE", true, false, false) PG_CMDTAG(CMDTAG_DROP_USER_MAPPING, "DROP USER MAPPING", true, false, false) +PG_CMDTAG(CMDTAG_DROP_VARIABLE, "DROP VARIABLE", true, false, false) PG_CMDTAG(CMDTAG_DROP_VIEW, "DROP VIEW", true, false, false) PG_CMDTAG(CMDTAG_EXECUTE, "EXECUTE", false, false, false) PG_CMDTAG(CMDTAG_EXPLAIN, "EXPLAIN", false, false, false) diff --git a/src/include/utils/acl.h b/src/include/utils/acl.h index f8e1238fa2..42e286a7ae 100644 --- a/src/include/utils/acl.h +++ b/src/include/utils/acl.h @@ -169,6 +169,7 @@ typedef struct ArrayType Acl; #define ACL_ALL_RIGHTS_SCHEMA (ACL_USAGE|ACL_CREATE) #define ACL_ALL_RIGHTS_TABLESPACE (ACL_CREATE) #define ACL_ALL_RIGHTS_TYPE (ACL_USAGE) +#define ACL_ALL_RIGHTS_VARIABLE (ACL_SELECT|ACL_UPDATE) /* operation codes for pg_*_aclmask */ typedef enum diff --git a/src/include/utils/lsyscache.h b/src/include/utils/lsyscache.h index 4f5418b972..283db9f725 100644 --- a/src/include/utils/lsyscache.h +++ b/src/include/utils/lsyscache.h @@ -132,6 +132,7 @@ extern char get_func_prokind(Oid funcid); extern bool get_func_leakproof(Oid funcid); extern RegProcedure get_func_support(Oid funcid); extern Oid get_relname_relid(const char *relname, Oid relnamespace); +extern Oid get_varname_varid(const char *varname, Oid varnamespace); extern char *get_rel_name(Oid relid); extern Oid get_rel_namespace(Oid relid); extern Oid get_rel_type_id(Oid relid); @@ -203,6 +204,14 @@ extern char *get_publication_name(Oid pubid, bool missing_ok); extern Oid get_subscription_oid(const char *subname, bool missing_ok); extern char *get_subscription_name(Oid subid, bool missing_ok); +extern char *get_session_variable_name(Oid varid); +extern Oid get_session_variable_namespace(Oid varid); +extern Oid get_session_variable_type(Oid varid); +extern void get_session_variable_type_typmod_collid(Oid varid, + Oid *typid, + int32 *typmod, + Oid *collid); + #define type_is_array(typid) (get_element_type(typid) != InvalidOid) /* type_is_array_domain accepts both plain arrays and domains over arrays */ #define type_is_array_domain(typid) (get_base_element_type(typid) != InvalidOid) diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h index 67ea6e4945..633c84c4d3 100644 --- a/src/include/utils/syscache.h +++ b/src/include/utils/syscache.h @@ -113,9 +113,11 @@ enum SysCacheIdentifier TYPENAMENSP, TYPEOID, USERMAPPINGOID, - USERMAPPINGUSERSERVER + USERMAPPINGUSERSERVER, + VARIABLENAMENSP, + VARIABLEOID -#define SysCacheSize (USERMAPPINGUSERSERVER + 1) +#define SysCacheSize (VARIABLEOID + 1) }; extern void InitCatalogCache(void); diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out index 215eb899be..d995332140 100644 --- a/src/test/regress/expected/oidjoins.out +++ b/src/test/regress/expected/oidjoins.out @@ -266,3 +266,7 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid} NOTICE: checking pg_subscription {subowner} => pg_authid {oid} NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid} NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid} +NOTICE: checking pg_variable {vartype} => pg_type {oid} +NOTICE: checking pg_variable {varnamespace} => pg_namespace {oid} +NOTICE: checking pg_variable {varowner} => pg_authid {oid} +NOTICE: checking pg_variable {varcollation} => pg_collation {oid} diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index b4058b88c3..88aa4a29c2 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -502,6 +502,7 @@ CreateRoleStmt CreateSchemaStmt CreateSchemaStmtContext CreateSeqStmt +CreateSessionVarStmt CreateStatsStmt CreateStmt CreateStmtContext @@ -830,6 +831,7 @@ FormData_pg_ts_parser FormData_pg_ts_template FormData_pg_type FormData_pg_user_mapping +FormData_pg_variable Form_pg_aggregate Form_pg_am Form_pg_amop @@ -888,6 +890,7 @@ Form_pg_ts_parser Form_pg_ts_template Form_pg_type Form_pg_user_mapping +Form_pg_variable FormatNode FreeBlockNumberArray FreeListData @@ -2662,6 +2665,8 @@ SupportRequestRows SupportRequestSelectivity SupportRequestSimplify SupportRequestWFuncMonotonic +SVariableXActAction +SVariableXActActionItem Syn SyncOps SyncRepConfigData -- 2.40.1 ^ permalink raw reply [nested|flat] 124+ messages in thread
* Re: Schema variables - new implementation for Postgres 15 @ 2023-05-17 03:20 Pavel Stehule <[email protected]> parent: Pavel Stehule <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Pavel Stehule @ 2023-05-17 03:20 UTC (permalink / raw) To: Dmitry Dolgov <[email protected]>; +Cc: Sergey Shinderuk <[email protected]>; Tomas Vondra <[email protected]>; Julien Rouhaud <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected] Hi út 16. 5. 2023 v 20:11 odesÃlatel Pavel Stehule <[email protected]> napsal: > Hi > > still in old layout - but it can be useful for testing by someone > > fix build doc, fix regress tests > fresh rebase > > Regards > > Pavel > Attachments: [text/x-patch] v20230517-1-0008-regress-tests-for-session-variables.patch (64.7K, ../../CAFj8pRAM60yANCnqKHHXZM3j++wpXWOu+rw6tVq5m531LHXiSw@mail.gmail.com/3-v20230517-1-0008-regress-tests-for-session-variables.patch) download | inline diff: From e08dd6f794e14fbe1fa013068ae9d9b5e10f02b0 Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Mon, 4 Apr 2022 20:54:43 +0200 Subject: [PATCH 08/10] regress tests for session variables Checks of basic functionality, check of usage session variables in subtransactions and from PLpgSQL --- .../isolation/expected/session-variable.out | 112 ++ src/test/isolation/isolation_schedule | 1 + .../isolation/specs/session-variable.spec | 53 + .../regress/expected/session_variables.out | 1564 +++++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/sql/session_variables.sql | 1203 +++++++++++++ 6 files changed, 2934 insertions(+), 1 deletion(-) create mode 100644 src/test/isolation/expected/session-variable.out create mode 100644 src/test/isolation/specs/session-variable.spec create mode 100644 src/test/regress/expected/session_variables.out create mode 100644 src/test/regress/sql/session_variables.sql diff --git a/src/test/isolation/expected/session-variable.out b/src/test/isolation/expected/session-variable.out new file mode 100644 index 0000000000..1c4149f6ef --- /dev/null +++ b/src/test/isolation/expected/session-variable.out @@ -0,0 +1,112 @@ +Parsed test spec with 4 sessions + +starting permutation: let val drop val +step let: LET myvar = 'test'; +step val: SELECT myvar; +myvar +----- +test +(1 row) + +step drop: DROP VARIABLE myvar; +step val: SELECT myvar; +ERROR: column or variable "myvar" does not exist + +starting permutation: let val s1 drop val sr1 +step let: LET myvar = 'test'; +step val: SELECT myvar; +myvar +----- +test +(1 row) + +step s1: BEGIN; +step drop: DROP VARIABLE myvar; +step val: SELECT myvar; +ERROR: column or variable "myvar" does not exist +step sr1: ROLLBACK; + +starting permutation: let val dbg drop create dbg val +step let: LET myvar = 'test'; +step val: SELECT myvar; +myvar +----- +test +(1 row) + +step dbg: SELECT schema, name, removed FROM pg_session_variables(); +schema|name |removed +------+-----+------- +public|myvar|f +(1 row) + +step drop: DROP VARIABLE myvar; +step create: CREATE VARIABLE myvar AS text; +step dbg: SELECT schema, name, removed FROM pg_session_variables(); +schema|name|removed +------+----+------- +(0 rows) + +step val: SELECT myvar; +myvar +----- + +(1 row) + + +starting permutation: let val s1 dbg drop create dbg val sr1 +step let: LET myvar = 'test'; +step val: SELECT myvar; +myvar +----- +test +(1 row) + +step s1: BEGIN; +step dbg: SELECT schema, name, removed FROM pg_session_variables(); +schema|name |removed +------+-----+------- +public|myvar|f +(1 row) + +step drop: DROP VARIABLE myvar; +step create: CREATE VARIABLE myvar AS text; +step dbg: SELECT schema, name, removed FROM pg_session_variables(); +schema|name |removed +------+-----+------- +public|myvar|f +(1 row) + +step val: SELECT myvar; +myvar +----- + +(1 row) + +step sr1: ROLLBACK; + +starting permutation: create3 let3 s3 o_c_d o_eox_r create4 let4 drop4 drop3 inval3 discard sc3 clean state +step create3: CREATE VARIABLE myvar3 AS text; +step let3: LET myvar3 = 'test'; +step s3: BEGIN; +step o_c_d: CREATE TEMP VARIABLE myvar_o_c_d AS text ON COMMIT DROP; +step o_eox_r: CREATE VARIABLE myvar_o_eox_r AS text ON TRANSACTION END RESET; LET myvar_o_eox_r = 'test'; +step create4: CREATE VARIABLE myvar4 AS text; +step let4: LET myvar4 = 'test'; +step drop4: DROP VARIABLE myvar4; +step drop3: DROP VARIABLE myvar3; +step inval3: SELECT COUNT(*) >= 0 FROM pg_foreign_table; +?column? +-------- +t +(1 row) + +step discard: DISCARD VARIABLES; +step sc3: COMMIT; +step clean: DROP VARIABLE myvar_o_eox_r; +step state: SELECT varname FROM pg_variable; +varname +------- +myvar +(1 row) + diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index 4fc56ae99c..809f47b941 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -110,3 +110,4 @@ test: serializable-parallel test: serializable-parallel-2 test: serializable-parallel-3 test: matview-write-skew +test: session-variable diff --git a/src/test/isolation/specs/session-variable.spec b/src/test/isolation/specs/session-variable.spec new file mode 100644 index 0000000000..5d089c8a4e --- /dev/null +++ b/src/test/isolation/specs/session-variable.spec @@ -0,0 +1,53 @@ +# Test session variables memory cleanup for sinval + +setup +{ + CREATE VARIABLE myvar AS text; +} + +teardown +{ + DROP VARIABLE IF EXISTS myvar; +} + +session s1 +step s1 { BEGIN; } +step let { LET myvar = 'test'; } +step val { SELECT myvar; } +step dbg { SELECT schema, name, removed FROM pg_session_variables(); } +step sr1 { ROLLBACK; } + +session s2 +step drop { DROP VARIABLE myvar; } +step create { CREATE VARIABLE myvar AS text; } + +session s3 +step s3 { BEGIN; } +step let3 { LET myvar3 = 'test'; } +step o_c_d { CREATE TEMP VARIABLE myvar_o_c_d AS text ON COMMIT DROP; } +step o_eox_r { CREATE VARIABLE myvar_o_eox_r AS text ON TRANSACTION END RESET; LET myvar_o_eox_r = 'test'; } +step create4 { CREATE VARIABLE myvar4 AS text; } +step let4 { LET myvar4 = 'test'; } +step drop4 { DROP VARIABLE myvar4; } +step inval3 { SELECT COUNT(*) >= 0 FROM pg_foreign_table; } +step discard { DISCARD VARIABLES; } +step sc3 { COMMIT; } +step clean { DROP VARIABLE myvar_o_eox_r; } +step state { SELECT varname FROM pg_variable; } + +session s4 +step create3 { CREATE VARIABLE myvar3 AS text; } +step drop3 { DROP VARIABLE myvar3; } + +# Concurrent drop of a known variable should lead to an error +permutation let val drop val +# Same, but with an explicit transaction +permutation let val s1 drop val sr1 +# Concurrent drop/create of a known variable should lead to empty variable +permutation let val dbg drop create dbg val +# Concurrent drop/create of a known variable should lead to empty variable +# We need a transaction to make sure that we won't accept invalidation when +# calling the dbg step after the concurrent drop +permutation let val s1 dbg drop create dbg val sr1 +# test for DISCARD ALL when all internal queues have actions registered +permutation create3 let3 s3 o_c_d o_eox_r create4 let4 drop4 drop3 inval3 discard sc3 clean state diff --git a/src/test/regress/expected/session_variables.out b/src/test/regress/expected/session_variables.out new file mode 100644 index 0000000000..463b2c96f6 --- /dev/null +++ b/src/test/regress/expected/session_variables.out @@ -0,0 +1,1564 @@ +-- should fail, pseudotypes are not allowed +CREATE VARIABLE xx AS anyelement; +ERROR: session variable cannot be pseudo-type anyelement +-- should be ok +CREATE SCHEMA svartest CREATE VARIABLE xx AS int CREATE TABLE foo(a int); +LET svartest.xx = 100; +DROP SCHEMA svartest CASCADE; +NOTICE: drop cascades to 2 other objects +DETAIL: drop cascades to table svartest.foo +drop cascades to session variable svartest.xx +-- should fail +LET svartest.xx = 100; +ERROR: session variable "svartest.xx" doesn't exist +LINE 1: LET svartest.xx = 100; + ^ +CREATE SCHEMA svartest; +SET search_path = svartest; +CREATE VARIABLE var1 AS integer; +CREATE TEMP VARIABLE var2 AS text; +DROP VARIABLE var1, var2; +-- functional interface +CREATE VARIABLE var1 AS numeric; +CREATE ROLE var_test_role; +GRANT USAGE ON SCHEMA svartest TO var_test_role; +SET ROLE TO var_test_role; +-- should fail +SELECT var1; +ERROR: permission denied for session variable var1 +SET ROLE TO DEFAULT; +GRANT SELECT ON VARIABLE var1 TO var_test_role; +SET ROLE TO var_test_role; +-- should fail +LET var1 = 10; +ERROR: permission denied for session variable var1 +-- should work +SELECT var1; + var1 +------ + +(1 row) + +SET ROLE TO DEFAULT; +GRANT UPDATE ON VARIABLE var1 TO var_test_role; +SET ROLE TO var_test_role; +-- should work +LET var1 = 333; +SET ROLE TO DEFAULT; +REVOKE ALL ON VARIABLE var1 FROM var_test_role; +CREATE OR REPLACE FUNCTION secure_var() +RETURNS int AS $$ + SELECT svartest.var1::int; +$$ LANGUAGE sql SECURITY DEFINER; +SELECT secure_var(); + secure_var +------------ + 333 +(1 row) + +SET ROLE TO var_test_role; +-- should fail +SELECT svartest.var1; +ERROR: permission denied for session variable var1 +-- should work; +SELECT secure_var(); + secure_var +------------ + 333 +(1 row) + +SET ROLE TO DEFAULT; +EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM generate_series(1,100) g(v) WHERE v = var1; + QUERY PLAN +----------------------------------------------- + Function Scan on pg_catalog.generate_series g + Output: v + Function Call: generate_series(1, 100) + Filter: ((g.v)::numeric = var1) +(4 rows) + +CREATE VIEW schema_var_view AS SELECT var1; +SELECT * FROM schema_var_view; + var1 +------ + 333 +(1 row) + +\c - +SET search_path = svartest; +-- should work still, but var will be empty +SELECT * FROM schema_var_view; + var1 +------ + +(1 row) + +LET var1 = pi(); +SELECT var1; + var1 +------------------ + 3.14159265358979 +(1 row) + +-- we can see execution plan of LET statement +EXPLAIN (VERBOSE, COSTS OFF) LET var1 = pi(); + QUERY PLAN +---------------------------- + SET SESSION VARIABLE + Result + Output: 3.14159265358979 +(3 rows) + +SELECT var1; + var1 +------------------ + 3.14159265358979 +(1 row) + +CREATE VARIABLE var3 AS int; +CREATE OR REPLACE FUNCTION inc(int) +RETURNS int AS $$ +BEGIN + LET svartest.var3 = COALESCE(svartest.var3 + $1, $1); + RETURN var3; +END; +$$ LANGUAGE plpgsql; +SELECT inc(1); + inc +----- + 1 +(1 row) + +SELECT inc(1); + inc +----- + 2 +(1 row) + +SELECT inc(1); + inc +----- + 3 +(1 row) + +SELECT inc(1) FROM generate_series(1,10); + inc +----- + 4 + 5 + 6 + 7 + 8 + 9 + 10 + 11 + 12 + 13 +(10 rows) + +SET ROLE TO var_test_role; +-- should fail +LET var3 = 0; +ERROR: permission denied for session variable var3 +SET ROLE TO DEFAULT; +DROP VIEW schema_var_view; +DROP VARIABLE var1 CASCADE; +DROP VARIABLE var3 CASCADE; +-- composite variables +CREATE TYPE sv_xyz AS (x int, y int, z numeric(10,2)); +CREATE VARIABLE v1 AS sv_xyz; +CREATE VARIABLE v2 AS sv_xyz; +\d v1 +\d v2 +LET v1 = (1,2,3.14); +LET v2 = (10,20,3.14*10); +-- should work too - there are prepared casts +LET v1 = (1,2,3.14); +SELECT v1; + v1 +------------ + (1,2,3.14) +(1 row) + +SELECT v2; + v2 +--------------- + (10,20,31.40) +(1 row) + +SELECT (v1).*; + x | y | z +---+---+------ + 1 | 2 | 3.14 +(1 row) + +SELECT (v2).*; + x | y | z +----+----+------- + 10 | 20 | 31.40 +(1 row) + +SELECT v1.x + v1.z; + ?column? +---------- + 4.14 +(1 row) + +SELECT v2.x + v2.z; + ?column? +---------- + 41.40 +(1 row) + +-- access to composite fields should be safe too +-- should fail +SET ROLE TO var_test_role; +SELECT v2.x; +ERROR: permission denied for session variable v2 +SET ROLE TO DEFAULT; +DROP VARIABLE v1; +DROP VARIABLE v2; +REVOKE USAGE ON SCHEMA svartest FROM var_test_role; +DROP ROLE var_test_role; +-- scalar variables should not be in conflict with qualified column +CREATE VARIABLE varx AS text; +SELECT varx.relname FROM pg_class varx WHERE varx.relname = 'pg_class'; + relname +---------- + pg_class +(1 row) + +-- should fail +SELECT varx.xxx; +ERROR: variable "svartest.varx" is of type "text", which is not a composite type +LINE 1: SELECT varx.xxx; + ^ +-- don't allow multi column query +CREATE TYPE vartesttp AS (a1 int, b1 int, c1 int); +CREATE VARIABLE v1 AS vartesttp; +-- should fail +LET v1 = (NULL::vartesttp).*; +ERROR: assignment expression returned 3 columns +LINE 1: LET v1 = (NULL::vartesttp).*; + ^ +DROP VARIABLE v1; +DROP TYPE vartesttp; +-- variables can be updated under RO transaction +BEGIN; +SET TRANSACTION READ ONLY; +LET varx = 'hello'; +COMMIT; +SELECT varx; + varx +------- + hello +(1 row) + +DROP VARIABLE varx; +CREATE TYPE t1 AS (a int, b numeric, c text); +CREATE VARIABLE v1 AS t1; +LET v1 = (1, pi(), 'hello'); +SELECT v1; + v1 +---------------------------- + (1,3.14159265358979,hello) +(1 row) + +LET v1.b = 10.2222; +SELECT v1; + v1 +------------------- + (1,10.2222,hello) +(1 row) + +-- should fail +LET v1.x = 10; +ERROR: cannot assign to field "x" of column "v1" because there is no such column in data type t1 +LINE 1: LET v1.x = 10; + ^ +DROP VARIABLE v1; +DROP TYPE t1; +-- arrays are supported +CREATE VARIABLE va1 AS numeric[]; +LET va1 = ARRAY[1.1,2.1]; +LET va1[1] = 10.1; +SELECT va1; + va1 +------------ + {10.1,2.1} +(1 row) + +-- LET target doesn't allow srf, should fail +LET va1[generate_series(1,3)] = 100; +ERROR: set-returning functions are not allowed in LET target expression +LINE 1: LET va1[generate_series(1,3)] = 100; + ^ +CREATE TYPE ta2 AS (a numeric, b numeric[]); +CREATE VARIABLE va2 AS ta2; +LET va2 = (10.1, ARRAY[0.0, 0.0]); +LET va2.a = 10.2; +SELECT va2; + va2 +-------------------- + (10.2,"{0.0,0.0}") +(1 row) + +LET va2.b[1] = 10.3; +SELECT va2; + va2 +--------------------- + (10.2,"{10.3,0.0}") +(1 row) + +DROP VARIABLE va1; +DROP VARIABLE va2; +DROP TYPE ta2; +-- default values +CREATE VARIABLE v1 AS numeric DEFAULT pi(); +LET v1 = v1 * 2; +SELECT v1; + v1 +------------------ + 6.28318530717958 +(1 row) + +CREATE TYPE t2 AS (a numeric, b text); +CREATE VARIABLE v2 AS t2 DEFAULT (NULL, 'Hello'); +LET svartest.v2.a = pi(); +SELECT v2; + v2 +-------------------------- + (3.14159265358979,Hello) +(1 row) + +-- should fail due dependency +DROP TYPE t2; +ERROR: cannot drop type t2 because other objects depend on it +DETAIL: session variable v2 depends on type t2 +HINT: Use DROP ... CASCADE to drop the dependent objects too. +-- should be ok +DROP VARIABLE v1; +DROP VARIABLE v2; +-- tests of alters +CREATE SCHEMA var_schema1; +CREATE SCHEMA var_schema2; +CREATE VARIABLE var_schema1.var1 AS integer; +LET var_schema1.var1 = 1000; +SELECT var_schema1.var1; + var1 +------ + 1000 +(1 row) + +ALTER VARIABLE var_schema1.var1 SET SCHEMA var_schema2; +SELECT var_schema2.var1; + var1 +------ + 1000 +(1 row) + +CREATE ROLE var_test_role; +ALTER VARIABLE var_schema2.var1 OWNER TO var_test_role; +SET ROLE TO var_test_role; +-- should fail, no access to schema var_schema2.var +SELECT var_schema2.var1; +ERROR: permission denied for schema var_schema2 +DROP VARIABLE var_schema2.var1; +ERROR: permission denied for schema var_schema2 +SET ROLE TO DEFAULT; +ALTER VARIABLE var_schema2.var1 SET SCHEMA public; +SET ROLE TO var_test_role; +SELECT public.var1; + var1 +------ + 1000 +(1 row) + +ALTER VARIABLE public.var1 RENAME TO var1_renamed; +SELECT public.var1_renamed; + var1_renamed +-------------- + 1000 +(1 row) + +DROP VARIABLE public.var1_renamed; +SET ROLE TO DEFAULt; +-- default rights test +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON VARIABLES TO var_test_role; +CREATE VARIABLE public.var2 AS int; +SET ROLE TO var_test_role; +-- should be ok +LET public.var2 = 100; +SELECT public.var2; + var2 +------ + 100 +(1 row) + +SET ROLE TO DEFAULt; +DROP VARIABLE public.var2; +DROP OWNED BY var_test_role; +DROP ROLE var_test_role; +CREATE VARIABLE xx AS text DEFAULT 'hello'; +SELECT xx, upper(xx); + xx | upper +-------+------- + hello | HELLO +(1 row) + +LET xx = 'Hi'; +SELECT xx; + xx +---- + Hi +(1 row) + +DROP VARIABLE xx; +-- ON TRANSACTION END RESET tests +CREATE VARIABLE t1 AS int DEFAULT -1 ON TRANSACTION END RESET; +BEGIN; + SELECT t1; + t1 +---- + -1 +(1 row) + + LET t1 = 100; + SELECT t1; + t1 +----- + 100 +(1 row) + +COMMIT; +SELECT t1; + t1 +---- + -1 +(1 row) + +BEGIN; + SELECT t1; + t1 +---- + -1 +(1 row) + + LET t1 = 100; + SELECT t1; + t1 +----- + 100 +(1 row) + +ROLLBACK; +SELECT t1; + t1 +---- + -1 +(1 row) + +DROP VARIABLE t1; +CREATE VARIABLE v1 AS int DEFAULT 0; +CREATE VARIABLE v2 AS text DEFAULT 'none'; +LET v1 = 100; +LET v2 = 'Hello'; +SELECT v1, v2; + v1 | v2 +-----+------- + 100 | Hello +(1 row) + +LET v1 = DEFAULT; +LET v2 = DEFAULT; +SELECT v1, v2; + v1 | v2 +----+------ + 0 | none +(1 row) + +DROP VARIABLE v1; +DROP VARIABLE v2; +-- ON COMMIT DROP tests +-- should be 0 always +SELECT count(*) FROM pg_variable; + count +------- + 0 +(1 row) + +CREATE TEMP VARIABLE g AS int ON COMMIT DROP; +SELECT count(*) FROM pg_variable; + count +------- + 0 +(1 row) + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; +COMMIT; +SELECT count(*) FROM pg_variable; + count +------- + 0 +(1 row) + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; +ROLLBACK; +SELECT count(*) FROM pg_variable; + count +------- + 0 +(1 row) + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; + LET g = 1; + DISCARD VARIABLES; +COMMIT; +SELECT count(*) FROM pg_variable; + count +------- + 0 +(1 row) + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; + LET g = 1; + DISCARD VARIABLES; +ROLLBACK; +SELECT count(*) FROM pg_variable; + count +------- + 0 +(1 row) + +-- Encourage use of parallel plans +SET parallel_setup_cost = 0; +SET parallel_tuple_cost = 0; +SET min_parallel_table_scan_size = 0; +SET max_parallel_workers_per_gather = 2; +-- test on query with workers +CREATE TABLE svar_test(a int); +INSERT INTO svar_test SELECT * FROM generate_series(1,1000); +ANALYZE svar_test; +CREATE VARIABLE zero int; +LET zero = 0; +-- parallel workers should be used +EXPLAIN (costs off) SELECT count(*) FROM svar_test WHERE a%10 = zero; + QUERY PLAN +-------------------------------------------- + Aggregate + -> Gather + Workers Planned: 2 + -> Parallel Seq Scan on svar_test + Filter: ((a % 10) = zero) +(5 rows) + +-- result should be 100 +SELECT count(*) FROM svar_test WHERE a%10 = zero; + count +------- + 100 +(1 row) + +LET zero = (SELECT count(*) FROM svar_test); +-- result should be 1000 +SELECT zero; + zero +------ + 1000 +(1 row) + +-- parallel workers should be used +EXPLAIN (costs off) LET zero = (SELECT count(*) FROM svar_test); + QUERY PLAN +---------------------------------------------------------- + SET SESSION VARIABLE + Result + InitPlan 1 (returns $1) + -> Finalize Aggregate + -> Gather + Workers Planned: 2 + -> Partial Aggregate + -> Parallel Seq Scan on svar_test +(8 rows) + +-- test for dependency on relation +CREATE VARIABLE v_table AS svar_test; +ALTER TABLE svar_test ALTER COLUMN a TYPE text; +ERROR: cannot alter table "svar_test" because session variable "svartest.v_table" uses it +DROP VARIABLE v_table; +DROP TABLE svar_test; +DROP VARIABLE zero; +RESET parallel_setup_cost; +RESET parallel_tuple_cost; +RESET min_parallel_table_scan_size; +RESET max_parallel_workers_per_gather; +-- use variables in prepared statements +CREATE VARIABLE v AS numeric; +LET v = 3.14; +-- use variables in views +CREATE VIEW vv AS SELECT COALESCE(v, 0) + 1000 AS result; +SELECT * FROM vv; + result +--------- + 1003.14 +(1 row) + +-- start a new session +\c +SET search_path to svartest; +SELECT * FROM vv; + result +-------- + 1000 +(1 row) + +LET v = 3.14; +SELECT * FROM vv; + result +--------- + 1003.14 +(1 row) + +-- should fail, dependency +DROP VARIABLE v; +ERROR: cannot drop session variable v because other objects depend on it +DETAIL: view vv depends on session variable v +HINT: Use DROP ... CASCADE to drop the dependent objects too. +-- should be ok +DROP VARIABLE v CASCADE; +NOTICE: drop cascades to view vv +-- other features +CREATE VARIABLE dt AS integer DEFAULT 0; +LET dt = 100; +SELECT dt; + dt +----- + 100 +(1 row) + +DISCARD VARIABLES; +SELECT dt; + dt +---- + 0 +(1 row) + +DROP VARIABLE dt; +-- NOT NULL +CREATE VARIABLE v1 AS int NOT NULL; +CREATE VARIABLE v2 AS int NOT NULL DEFAULT NULL; +-- should fail +SELECT v1; +ERROR: null value is not allowed for NOT NULL session variable "svartest.v1" +DETAIL: The session variable was not initialized yet. +SELECT v2; +ERROR: null value is not allowed for NOT NULL session variable "svartest.v2" +LET v1 = NULL; +ERROR: null value is not allowed for NOT NULL session variable "svartest.v1" +LET v2 = NULL; +ERROR: null value is not allowed for NOT NULL session variable "svartest.v2" +LET v1 = DEFAULT; +ERROR: null value is not allowed for NOT NULL session variable "svartest.v1" +LET v2 = DEFAULT; +ERROR: null value is not allowed for NOT NULL session variable "svartest.v2" +-- should be ok +LET v1 = 100; +LET v2 = 1000; +SELECT v1, v2; + v1 | v2 +-----+------ + 100 | 1000 +(1 row) + +DROP VARIABLE v1; +DROP VARIABLE v2; +CREATE VARIABLE tv AS int; +CREATE VARIABLE IF NOT EXISTS tv AS int; +NOTICE: session variable "tv" already exists, skipping +DROP VARIABLE tv; +CREATE IMMUTABLE VARIABLE iv AS int DEFAULT 100; +SELECT iv; + iv +----- + 100 +(1 row) + +-- should fail; +LET iv = 10000; +ERROR: session variable "svartest.iv" is declared IMMUTABLE +DROP VARIABLE iv; +-- different order +CREATE IMMUTABLE VARIABLE iv AS int DEFAULT 100; +-- should to fail +LET iv = 10000; +ERROR: session variable "svartest.iv" is declared IMMUTABLE +-- should be ok +SELECT iv; + iv +----- + 100 +(1 row) + +DROP VARIABLE iv; +CREATE IMMUTABLE VARIABLE iv AS int; +-- should be ok +LET iv = NULL; +-- should fail +LET iv = NULL; +ERROR: session variable "svartest.iv" is declared IMMUTABLE +DROP VARIABLE iv; +-- create variable inside plpgsql block +DO $$ +BEGIN + CREATE VARIABLE do_test_svar AS date DEFAULT '2000-01-01'; +END; +$$; +SELECT do_test_svar; + do_test_svar +-------------- + 01-01-2000 +(1 row) + +DROP VARIABLE do_test_svar; +-- should fail +CREATE IMMUTABLE VARIABLE xx AS int NOT NULL; +ERROR: IMMUTABLE NOT NULL variable requires default expression +-- REASSIGN OWNED test +CREATE ROLE var_test_role1; +CREATE ROLE var_test_role2; +CREATE VARIABLE xxx_var AS int; +ALTER VARIABLE xxx_var OWNER TO var_test_role1; +REASSIGN OWNED BY var_test_role1 to var_test_role2; +SELECT varowner::regrole FROM pg_variable WHERE varname = 'xxx_var'; + varowner +---------------- + var_test_role2 +(1 row) + +DROP OWNED BY var_test_role1; +DROP ROLE var_test_role1; +SELECT count(*) FROM pg_variable WHERE varname = 'xxx_var'; + count +------- + 1 +(1 row) + +DROP OWNED BY var_test_role2; +DROP ROLE var_test_role2; +SELECT count(*) FROM pg_variable WHERE varname = 'xxx_var'; + count +------- + 0 +(1 row) + +-- creating, dropping temporary variable +BEGIN; +CREATE TEMP VARIABLE tempvar AS INT ON COMMIT DROP; +LET tempvar = 100; +SAVEPOINT s1; +DROP VARIABLE tempvar; +ROLLBACK TO s1; +SELECT tempvar; + tempvar +--------- + 100 +(1 row) + +COMMIT; +-- should to fail +LET tempvar = 100; +ERROR: session variable "tempvar" doesn't exist +LINE 1: LET tempvar = 100; + ^ +BEGIN; +SAVEPOINT s1; +CREATE TEMP VARIABLE tempvar AS INT ON COMMIT DROP; +LET tempvar = 100; +ROLLBACK TO s1; +COMMIT; +-- should to fail +LET tempvar = 100; +ERROR: session variable "tempvar" doesn't exist +LINE 1: LET tempvar = 100; + ^ +CREATE VARIABLE var1 AS int; +LET var1 = 100; +BEGIN; +DROP VARIABLE var1; +ROLLBACK; +SELECT var1; + var1 +------ + 100 +(1 row) + +DROP VARIABLE var1; +CREATE VARIABLE var1 AS int DEFAULT 100; +COMMENT ON VARIABLE var1 IS 'some variable comment'; +SELECT pg_catalog.obj_description(oid, 'pg_variable') FROM pg_variable WHERE varname = 'var1'; + obj_description +----------------------- + some variable comment +(1 row) + +DROP VARIABLE var1; +CREATE TABLE xxtab(avar int); +CREATE TYPE xxtype AS (avar int); +CREATE VARIABLE xxtab AS xxtype; +INSERT INTO xxtab VALUES(10); +-- it is ambiguous, but columns are preferred +-- variable is shadowed +SELECT xxtab.avar FROM xxtab; + avar +------ + 10 +(1 row) + +SET session_variables_ambiguity_warning TO on; +SELECT xxtab.avar FROM xxtab; +WARNING: session variable "xxtab.avar" is shadowed +LINE 1: SELECT xxtab.avar FROM xxtab; + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. + avar +------ + 10 +(1 row) + +SET search_path = svartest; +CREATE VARIABLE testvar as int; +-- plpgsql variables are preferred against session variables +DO $$ +<<myblock>> +DECLARE testvar int; +BEGIN + -- should be ok without warning + LET testvar = 100; + -- should be ok without warning + testvar := 1000; + -- should be ok without warning + RAISE NOTICE 'session variable is %', svartest.testvar; + -- should be ok without warning + RAISE NOTICE 'plpgsql variable is %', myblock.testvar; + -- should to print plpgsql variable with warning + RAISE NOTICE 'variable is %', testvar; +END; +$$; +WARNING: session variable "testvar" is shadowed +LINE 1: testvar := 1000 + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. +QUERY: testvar := 1000 +NOTICE: session variable is 100 +NOTICE: plpgsql variable is 1000 +WARNING: session variable "testvar" is shadowed +LINE 1: testvar + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. +QUERY: testvar +NOTICE: variable is 1000 +DROP VARIABLE testvar; +SET session_variables_ambiguity_warning TO default; +-- should be ok +SELECT avar FROM xxtab; + avar +------ + 10 +(1 row) + +CREATE VARIABLE public.avar AS int; +-- should be ok, see the table +SELECT avar FROM xxtab; + avar +------ + 10 +(1 row) + +-- should be ok +SELECT public.avar FROM xxtab; + avar +------ + +(1 row) + +DROP VARIABLE xxtab; +SELECT xxtab.avar FROM xxtab; + avar +------ + 10 +(1 row) + +DROP VARIABLE public.avar; +DROP TYPE xxtype; +DROP TABLE xxtab; +-- test of plan cache invalidation +CREATE VARIABLE xx AS int; +SET plan_cache_mode = force_generic_plan; +PREPARE pp AS SELECT xx; +EXECUTE pp; + xx +---- + +(1 row) + +DROP VARIABLE xx; +CREATE VARIABLE xx AS int; +-- should to work +EXECUTE pp; + xx +---- + +(1 row) + +DROP VARIABLE xx; +DEALLOCATE pp; +SET plan_cache_mode = DEFAULT; +CREATE ROLE var_test_role; +CREATE SCHEMA vartest; +GRANT USAGE ON SCHEMA vartest TO var_test_role; +CREATE VARIABLE vartest.x AS int; +CREATE VARIABLE vartest.y AS int; +LET vartest.x = 100; +LET vartest.y = 101; +GRANT SELECT ON ALL VARIABLES IN SCHEMA vartest TO var_test_role; +SET ROLE TO var_test_role; +SELECT vartest.x, vartest.y; + x | y +-----+----- + 100 | 101 +(1 row) + +SET ROLE TO DEFAULT; +REVOKE SELECT ON ALL VARIABLES IN SCHEMA vartest FROM var_test_role; +SET ROLE TO var_test_role; +-- should to fail +SELECT vartest.x; +ERROR: permission denied for session variable x +SELECT vartest.y; +ERROR: permission denied for session variable y +SET ROLE TO DEFAULT; +DROP VARIABLE vartest.x, vartest.y; +DROP SCHEMA vartest; +DROP ROLE var_test_role; +-- test cached plan +CREATE VARIABLE v1 AS text; +CREATE VARIABLE v2 AS int; +CREATE VARIABLE v3 AS int; +LET v1 = 'test'; +LET v2 = 10; +LET v3 = 5; +PREPARE q1 AS SELECT v1 || i FROM generate_series(1, v2) g(i) WHERE i IN (v2, v3); +SET plan_cache_mode to force_generic_plan; +EXECUTE q1; + ?column? +---------- + test5 + test10 +(2 rows) + +EXPLAIN EXECUTE q1; + QUERY PLAN +----------------------------------------------------------------------- + Function Scan on generate_series g (cost=0.00..0.14 rows=2 width=32) + Filter: (i = ANY (ARRAY[v2, v3])) +(2 rows) + +-- dependecy check +DROP VARIABLE v3; +-- recreate v3 again +CREATE VARIABLE v3 AS int DEFAULT 6; +-- should to work, the plan should be recreated +EXECUTE q1; + ?column? +---------- + test6 + test10 +(2 rows) + +DEALLOCATE q1; +-- fill v1 by long text +LET v1 = repeat(' ', 10000); +PREPARE q1 AS SELECT length(v1); +EXECUTE q1; + length +-------- + 10000 +(1 row) + +LET v1 = repeat(' ', 5000); +EXECUTE q1; + length +-------- + 5000 +(1 row) + +DEALLOCATE q1; +SET plan_cache_mode to default; +DROP VARIABLE v1, v2, v3; +CREATE ROLE var_test_role; +CREATE VARIABLE public.v1 AS int DEFAULT 0; +-- check acl when variable is acessed by simple eval expr method +CREATE OR REPLACE FUNCTION public.fx_var(int) +RETURNS int AS $$ +DECLARE xx int; +BEGIN + xx := public.v1 + $1; + RETURN xx; +END; +$$ LANGUAGE plpgsql; +-- should be ok +SELECT public.fx_var(0); + fx_var +-------- + 0 +(1 row) + +SET ROLE TO var_test_role; +-- should to fail +SELECT public.fx_var(0); +ERROR: permission denied for session variable v1 +CONTEXT: PL/pgSQL function public.fx_var(integer) line 4 at assignment +SET ROLE TO default; +GRANT SELECT ON VARIABLE public.v1 TO var_test_role; +SET ROLE TO var_test_role; +-- should be ok +SELECT public.fx_var(0); + fx_var +-------- + 0 +(1 row) + +SET ROLE TO default; +REVOKE SELECT ON VARIABLE public.v1 FROM var_test_role; +SET ROLE TO var_test_role; +-- should be fail +SELECT public.fx_var(0); +ERROR: permission denied for session variable v1 +CONTEXT: PL/pgSQL function public.fx_var(integer) line 4 at assignment +SET ROLE TO DEFAULT; +DROP FUNCTION public.fx_var(int); +DROP VARIABLE public.v1; +DROP ROLE var_test_role; +CREATE TYPE public.svar_test_type AS (a int, b int, c numeric); +CREATE VARIABLE public.svar AS public.svar_test_type; +LET public.svar = ROW(10,20,30); +SELECT public.svar; + svar +------------ + (10,20,30) +(1 row) + +ALTER TYPE public.svar_test_type DROP ATTRIBUTE c; +-- should be ok +SELECT public.svar; + svar +--------- + (10,20) +(1 row) + +ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; +-- should be ok +SELECT public.svar; + svar +---------- + (10,20,) +(1 row) + +LET public.svar = ROW(10,20,30); +-- should be ok again for new value +SELECT public.svar; + svar +------------ + (10,20,30) +(1 row) + +DROP VARIABLE public.svar; +DROP TYPE public.svar_test_type; +CREATE VARIABLE public.svar AS int; +select oid as varid from pg_variable where varname = 'svar' and varnamespace = 'public'::regnamespace \gset +SELECT schema, name, removed FROM pg_session_variables(); + schema | name | removed +--------+------+--------- +(0 rows) + +LET public.svar = 100; +SELECT schema, name, removed FROM pg_session_variables(); + schema | name | removed +--------+------+--------- + public | svar | f +(1 row) + +BEGIN; +DROP VARIABLE public.svar; +-- value should be in memory +SELECT schema, removed FROM pg_session_variables(); + schema | removed +--------+--------- + | t +(1 row) + +ROLLBACK; +-- value should be in memory +SELECT schema, name, removed FROM pg_session_variables() f WHERE f.varid = :varid; + schema | name | removed +--------+------+--------- + public | svar | f +(1 row) + +SELECT public.svar; + svar +------ + 100 +(1 row) + +BEGIN; +DROP VARIABLE public.svar; +-- value should be in memory +SELECT schema, name = :varid::text AS ok, removed FROM pg_session_variables() f WHERE f.varid = :varid;; + schema | ok | removed +--------+----+--------- + | t | t +(1 row) + +COMMIT; +-- the memory should be clean; +SELECT schema, name, removed FROM pg_session_variables() WHERE f.varid = :varid; +ERROR: missing FROM-clause entry for table "f" +LINE 1: ..., name, removed FROM pg_session_variables() WHERE f.varid = ... + ^ +BEGIN; +CREATE VARIABLE public.svar AS int; +select oid as varid from pg_variable where varname = 'svar' and varnamespace = 'public'::regnamespace \gset +LET public.svar = 100; +ROLLBACK; +-- the memory should be clean; +SELECT schema, name, removed FROM pg_session_variables() f WHERE f.varid = :varid; + schema | name | removed +--------+------+--------- +(0 rows) + +\unset varid +CREATE VARIABLE public.svar AS int; +LET public.svar = 100; +-- repeated aborted transaction +BEGIN; DROP VARIABLE public.svar; ROLLBACK; +BEGIN; DROP VARIABLE public.svar; ROLLBACK; +BEGIN; DROP VARIABLE public.svar; ROLLBACK; +-- the value should be still available +SELECT public.svar; + svar +------ + 100 +(1 row) + +DROP VARIABLE public.svar; +CREATE TYPE public.svar_test_type AS (a int, b int); +CREATE VARIABLE public.svar AS public.svar_test_type; +SELECT public.svar; + svar +------ + +(1 row) + +ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; +SELECT public.svar; + svar +------ + +(1 row) + +ALTER TYPE public.svar_test_type DROP ATTRIBUTE b; +SELECT public.svar; + svar +------ + +(1 row) + +DROP VARIABLE public.svar; +DROP TYPE public.svar_test_type; +CREATE TYPE public.svar_test_type AS (a int, b int); +CREATE VARIABLE public.svar AS public.svar_test_type; +CREATE VARIABLE public.svar2 AS public.svar_test_type; +LET public.svar = (10, 20); +ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; +SELECT public.svar; + svar +---------- + (10,20,) +(1 row) + +LET public.svar2 = (10, 20, 30); +ALTER TYPE public.svar_test_type DROP ATTRIBUTE b; +SELECT public.svar; + svar +------- + (10,) +(1 row) + +SELECT public.svar2; + svar2 +--------- + (10,30) +(1 row) + +DROP VARIABLE public.svar; +DROP VARIABLE public.svar2; +DROP TYPE public.svar_test_type; +-- The composite type cannot be changed when it is used +CREATE TYPE public.svar_type AS (a int, b int); +CREATE VARIABLE public.svar AS public.svar_type; +-- should to fail +ALTER TYPE public.svar_type ALTER ATTRIBUTE b TYPE numeric; +ERROR: cannot alter type "svar_type" because session variable "public.svar" uses it +DROP VARIABLE public.svar; +CREATE TYPE public.svar_type2 AS (a int, b int, c public.svar_type); +CREATE VARIABLE public.svar AS public.svar_type2; +-- should to fail +ALTER TYPE public.svar_type ALTER ATTRIBUTE b TYPE numeric; +ERROR: cannot alter type "svar_type" because session variable "public.svar" uses it +DROP VARIABLE public.svar; +DROP TYPE public.svar_type2; +DROP TYPE public.svar_type; +-- The variable can be shadowed by table or by alias +CREATE TYPE public.svar_type AS (a int, b int, c int); +CREATE VARIABLE public.svar AS public.svar_type; +CREATE TABLE public.svar (a int, b int); +INSERT INTO public.svar VALUES(10, 20); +LET public.svar = (100, 200, 300); +-- should be ok +-- show table +SELECT * FROM public.svar; + a | b +----+---- + 10 | 20 +(1 row) + +SELECT svar.a FROM public.svar; + a +---- + 10 +(1 row) + +SELECT svar.* FROM public.svar; + a | b +----+---- + 10 | 20 +(1 row) + +-- show variable +SELECT public.svar; + svar +--------------- + (100,200,300) +(1 row) + +SELECT public.svar.c; + c +----- + 300 +(1 row) + +SELECT (public.svar).*; + a | b | c +-----+-----+----- + 100 | 200 | 300 +(1 row) + +-- the variable is shadowed, raise error +SELECT public.svar.c FROM public.svar; +ERROR: column svar.c does not exist +LINE 1: SELECT public.svar.c FROM public.svar; + ^ +-- can be fixed by alias +SELECT public.svar.c FROM public.svar x; + c +----- + 300 +(1 row) + +-- again with warnings +SET session_variables_ambiguity_warning TO on; +SELECT * FROM public.svar; + a | b +----+---- + 10 | 20 +(1 row) + +SELECT svar.a FROM public.svar; + a +---- + 10 +(1 row) + +SELECT svar.* FROM public.svar; + a | b +----+---- + 10 | 20 +(1 row) + +-- show variable +SELECT public.svar; + svar +--------------- + (100,200,300) +(1 row) + +SELECT public.svar.c; + c +----- + 300 +(1 row) + +SELECT (public.svar).*; + a | b | c +-----+-----+----- + 100 | 200 | 300 +(1 row) + +-- the variable is shadowed, raise error +SELECT public.svar.c FROM public.svar; +WARNING: session variable "public.svar" is shadowed +LINE 1: SELECT public.svar.c FROM public.svar; + ^ +DETAIL: Session variables can be shadowed by tables or table's aliases with the same name. +ERROR: column svar.c does not exist +LINE 1: SELECT public.svar.c FROM public.svar; + ^ +-- can be fixed by alias +SELECT public.svar.c FROM public.svar x; + c +----- + 300 +(1 row) + +SET session_variables_ambiguity_warning TO DEFAULT; +DROP VARIABLE public.svar; +DROP TABLE public.svar; +DROP TYPE public.svar_type; +-- the LET statement should be disallowed in CTE +CREATE VARIABLE public.svar AS int; +WITH x AS (LET public.svar = 100) SELECT * FROM x; +ERROR: LET not supported in WITH query +LINE 1: WITH x AS (LET public.svar = 100) SELECT * FROM x; + ^ +-- the LET statement can be prepared +PREPARE pp(int) AS LET public.svar = $1; +EXECUTE pp(10); +SELECT public.svar; + svar +------ + 10 +(1 row) + +EXECUTE pp(1000); +SELECT public.svar; + svar +------ + 1000 +(1 row) + +-- test of correct dependency +DROP VARIABLE public.svar; +EXECUTE pp(-1); +ERROR: session variable "public.svar" doesn't exist +DEALLOCATE pp; +CREATE VARIABLE public.svar2 AS int; +CREATE VARIABLE public.svar1 AS int DEFAULT 100; +PREPARE pp(int) AS LET public.svar2 = public.svar1 + $1; +EXECUTE pp(100); +SELECT public.svar1, public.svar2; + svar1 | svar2 +-------+------- + 100 | 200 +(1 row) + +LET public.svar1 = 1000; +EXECUTE pp(100); +SELECT public.svar1, public.svar2; + svar1 | svar2 +-------+------- + 1000 | 1100 +(1 row) + +DROP VARIABLE public.svar1; +EXECUTE pp(100); +ERROR: missing FROM-clause entry for table "public" +DEALLOCATE pp; +DROP VARIABLE public.svar2; +CREATE VARIABLE public.svar AS int; +-- should be ok +LET public.svar = generate_series(1, 1); +-- should fail +LET public.svar = generate_series(1, 2); +ERROR: expression returned more than one row +LET public.svar = generate_series(1, 0); +ERROR: expression returned no rows +DROP VARIABLE public.svar; +SET search_path TO DEFAULT; +CREATE TYPE ab AS (a integer, b integer); +CREATE VARIABLE v_ab AS ab; +CREATE TABLE v_ab (a integer, b integer); +SET session_variables_ambiguity_warning = 1; +-- warning should be raised +SELECT v_ab.a FROM v_ab; +WARNING: session variable "v_ab.a" is shadowed +LINE 1: SELECT v_ab.a FROM v_ab; + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. + a +--- +(0 rows) + +CREATE SCHEMA v_ab; +CREATE VARIABLE v_ab.a AS integer; +-- warning should be raised +SELECT v_ab.a FROM v_ab; +WARNING: session variable "v_ab.a" is shadowed +LINE 1: SELECT v_ab.a FROM v_ab; + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. + a +--- +(0 rows) + +DROP VARIABLE v_ab; +DROP TABLE v_ab; +DROP TYPE ab; +CREATE VARIABLE myvar AS int; +SELECT myvar.blabla; +ERROR: variable "public.myvar" is of type "integer", which is not a composite type +LINE 1: SELECT myvar.blabla; + ^ +DROP VARIABLE myvar; +-- the result of view should be same in parallel mode too +CREATE VARIABLE v1 AS int; +LET v1 = 10; +CREATE VIEW v1view AS SELECT COALESCE(v1, 0) AS result; +SELECT * FROM v1view; + result +-------- + 10 +(1 row) + +SET debug_parallel_query TO on; +SELECT * FROM v1view; + result +-------- + 10 +(1 row) + +SET debug_parallel_query TO off; +DROP VIEW v1view; +DROP VARIABLE v1; +-- the value should not be corrupted +CREATE VARIABLE v1 text DEFAULT 'abc'; +CREATE FUNCTION ffunc() +RETURNS text AS $$ +BEGIN + RETURN gfunc(v1); +END +$$ LANGUAGE plpgsql; +CREATE FUNCTION gfunc(t text) +RETURNS text AS $$ +BEGIN + LET v1 = 'BOOM!'; + RETURN t; +END; +$$ LANGUAGE plpgsql; +select ffunc(); + ffunc +------- + abc +(1 row) + +DROP FUNCTION ffunc(); +DROP FUNCTION gfunc(text); +DROP VARIABLE v1; +CREATE TYPE t_am_type AS (b int); +CREATE SCHEMA xxx_am; +SET search_path TO public; +CREATE VARIABLE xxx_am AS t_am_type DEFAULT row(10); +SELECT xxx_am; + xxx_am +-------- + (10) +(1 row) + +SELECT public.xxx_am; + xxx_am +-------- + (10) +(1 row) + +CREATE VARIABLE xxx_am.b AS int DEFAULT 20; +-- the reference should be ambiguous +SELECT xxx_am.b; +ERROR: session variable reference "xxx_am.b" is ambiguous +LINE 1: SELECT xxx_am.b; + ^ +-- enhanced references should be ok +SELECT public.xxx_am.b; + b +---- + 10 +(1 row) + +SELECT :"DBNAME".xxx_am.b; + b +---- + 20 +(1 row) + +SET session_variables_ambiguity_warning TO on; +CREATE TABLE xxx_am(b int); +-- should be warning, not error +SELECT xxx_am.b FROM xxx_am; +WARNING: session variable "xxx_am.b" is shadowed +LINE 1: SELECT xxx_am.b FROM xxx_am; + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. + b +--- +(0 rows) + +-- no warning +SELECT x.b FROM xxx_am x; + b +--- +(0 rows) + +DROP TABLE xxx_am; +DROP VARIABLE public.xxx_am; +DROP VARIABLE xxx_am.b; +DROP SCHEMA xxx_am; +CREATE SCHEMA :"DBNAME"; +CREATE VARIABLE :"DBNAME".:"DBNAME".:"DBNAME" AS t_am_type DEFAULT row(10); +CREATE VARIABLE :"DBNAME".:"DBNAME".b AS int DEFAULT 20; +SET search_path to :"DBNAME"; +-- In this test case, error (and warning) messages contains database name. +-- It is "regression", but it can be possibly different, so we try to +-- supress error and warning messages. +\set ECHO none +ERROR: session variable reference "regression.b" is ambiguous +LINE 1: SELECT "regression".b; + ^ +true, 42P08 +ERROR: session variable reference "regression.regression.b" is ambiguous +LINE 1: SELECT "regression"."regression".b; + ^ +true, 42P08 + b +--- +(0 rows) + +false, 00000 +DROP TABLE :"DBNAME"; +DROP VARIABLE :"DBNAME".:"DBNAME".b; +DROP VARIABLE :"DBNAME".:"DBNAME".:"DBNAME"; +DROP SCHEMA :"DBNAME"; +SET client_min_messages TO DEFAULT; +SET search_path to public; diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index cf46fa3359..a45cd40a66 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -111,7 +111,7 @@ test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson # NB: temp.sql does a reconnect which transiently uses 2 connections, # so keep this parallel group to at most 19 tests # ---------- -test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion truncate alter_table sequence polymorphism rowtypes returning largeobject with xml +test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion truncate alter_table sequence polymorphism rowtypes returning largeobject with xml session_variables # ---------- # Another group of parallel tests diff --git a/src/test/regress/sql/session_variables.sql b/src/test/regress/sql/session_variables.sql new file mode 100644 index 0000000000..42926dfbcc --- /dev/null +++ b/src/test/regress/sql/session_variables.sql @@ -0,0 +1,1203 @@ +-- should fail, pseudotypes are not allowed +CREATE VARIABLE xx AS anyelement; + +-- should be ok +CREATE SCHEMA svartest CREATE VARIABLE xx AS int CREATE TABLE foo(a int); + +LET svartest.xx = 100; + +DROP SCHEMA svartest CASCADE; + +-- should fail +LET svartest.xx = 100; + +CREATE SCHEMA svartest; + +SET search_path = svartest; + +CREATE VARIABLE var1 AS integer; +CREATE TEMP VARIABLE var2 AS text; + +DROP VARIABLE var1, var2; + +-- functional interface +CREATE VARIABLE var1 AS numeric; + +CREATE ROLE var_test_role; +GRANT USAGE ON SCHEMA svartest TO var_test_role; + +SET ROLE TO var_test_role; + +-- should fail +SELECT var1; + +SET ROLE TO DEFAULT; + +GRANT SELECT ON VARIABLE var1 TO var_test_role; + +SET ROLE TO var_test_role; +-- should fail +LET var1 = 10; +-- should work +SELECT var1; + +SET ROLE TO DEFAULT; + +GRANT UPDATE ON VARIABLE var1 TO var_test_role; + +SET ROLE TO var_test_role; + +-- should work +LET var1 = 333; + +SET ROLE TO DEFAULT; + +REVOKE ALL ON VARIABLE var1 FROM var_test_role; + +CREATE OR REPLACE FUNCTION secure_var() +RETURNS int AS $$ + SELECT svartest.var1::int; +$$ LANGUAGE sql SECURITY DEFINER; + +SELECT secure_var(); + +SET ROLE TO var_test_role; + +-- should fail +SELECT svartest.var1; + +-- should work; +SELECT secure_var(); + +SET ROLE TO DEFAULT; + +EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM generate_series(1,100) g(v) WHERE v = var1; + +CREATE VIEW schema_var_view AS SELECT var1; + +SELECT * FROM schema_var_view; + +\c - + +SET search_path = svartest; + +-- should work still, but var will be empty +SELECT * FROM schema_var_view; + +LET var1 = pi(); + +SELECT var1; + +-- we can see execution plan of LET statement +EXPLAIN (VERBOSE, COSTS OFF) LET var1 = pi(); + +SELECT var1; + +CREATE VARIABLE var3 AS int; + +CREATE OR REPLACE FUNCTION inc(int) +RETURNS int AS $$ +BEGIN + LET svartest.var3 = COALESCE(svartest.var3 + $1, $1); + RETURN var3; +END; +$$ LANGUAGE plpgsql; + +SELECT inc(1); +SELECT inc(1); +SELECT inc(1); + +SELECT inc(1) FROM generate_series(1,10); + +SET ROLE TO var_test_role; + +-- should fail +LET var3 = 0; + +SET ROLE TO DEFAULT; + +DROP VIEW schema_var_view; + +DROP VARIABLE var1 CASCADE; +DROP VARIABLE var3 CASCADE; + +-- composite variables + +CREATE TYPE sv_xyz AS (x int, y int, z numeric(10,2)); + +CREATE VARIABLE v1 AS sv_xyz; +CREATE VARIABLE v2 AS sv_xyz; + +\d v1 +\d v2 + +LET v1 = (1,2,3.14); +LET v2 = (10,20,3.14*10); + +-- should work too - there are prepared casts +LET v1 = (1,2,3.14); + +SELECT v1; +SELECT v2; +SELECT (v1).*; +SELECT (v2).*; + +SELECT v1.x + v1.z; +SELECT v2.x + v2.z; + +-- access to composite fields should be safe too +-- should fail +SET ROLE TO var_test_role; + +SELECT v2.x; + +SET ROLE TO DEFAULT; + +DROP VARIABLE v1; +DROP VARIABLE v2; + +REVOKE USAGE ON SCHEMA svartest FROM var_test_role; +DROP ROLE var_test_role; + +-- scalar variables should not be in conflict with qualified column +CREATE VARIABLE varx AS text; +SELECT varx.relname FROM pg_class varx WHERE varx.relname = 'pg_class'; + +-- should fail +SELECT varx.xxx; + +-- don't allow multi column query +CREATE TYPE vartesttp AS (a1 int, b1 int, c1 int); +CREATE VARIABLE v1 AS vartesttp; + +-- should fail +LET v1 = (NULL::vartesttp).*; + +DROP VARIABLE v1; +DROP TYPE vartesttp; + +-- variables can be updated under RO transaction + +BEGIN; +SET TRANSACTION READ ONLY; +LET varx = 'hello'; +COMMIT; + +SELECT varx; + +DROP VARIABLE varx; + +CREATE TYPE t1 AS (a int, b numeric, c text); + +CREATE VARIABLE v1 AS t1; +LET v1 = (1, pi(), 'hello'); +SELECT v1; +LET v1.b = 10.2222; +SELECT v1; + +-- should fail +LET v1.x = 10; + +DROP VARIABLE v1; +DROP TYPE t1; + +-- arrays are supported +CREATE VARIABLE va1 AS numeric[]; +LET va1 = ARRAY[1.1,2.1]; +LET va1[1] = 10.1; +SELECT va1; + +-- LET target doesn't allow srf, should fail +LET va1[generate_series(1,3)] = 100; + +CREATE TYPE ta2 AS (a numeric, b numeric[]); +CREATE VARIABLE va2 AS ta2; +LET va2 = (10.1, ARRAY[0.0, 0.0]); +LET va2.a = 10.2; +SELECT va2; +LET va2.b[1] = 10.3; +SELECT va2; + +DROP VARIABLE va1; +DROP VARIABLE va2; +DROP TYPE ta2; + +-- default values +CREATE VARIABLE v1 AS numeric DEFAULT pi(); +LET v1 = v1 * 2; +SELECT v1; + +CREATE TYPE t2 AS (a numeric, b text); +CREATE VARIABLE v2 AS t2 DEFAULT (NULL, 'Hello'); +LET svartest.v2.a = pi(); +SELECT v2; + +-- should fail due dependency +DROP TYPE t2; + +-- should be ok +DROP VARIABLE v1; +DROP VARIABLE v2; + +-- tests of alters +CREATE SCHEMA var_schema1; +CREATE SCHEMA var_schema2; + +CREATE VARIABLE var_schema1.var1 AS integer; +LET var_schema1.var1 = 1000; +SELECT var_schema1.var1; +ALTER VARIABLE var_schema1.var1 SET SCHEMA var_schema2; +SELECT var_schema2.var1; + +CREATE ROLE var_test_role; + +ALTER VARIABLE var_schema2.var1 OWNER TO var_test_role; +SET ROLE TO var_test_role; + +-- should fail, no access to schema var_schema2.var +SELECT var_schema2.var1; +DROP VARIABLE var_schema2.var1; + +SET ROLE TO DEFAULT; + +ALTER VARIABLE var_schema2.var1 SET SCHEMA public; + +SET ROLE TO var_test_role; +SELECT public.var1; + +ALTER VARIABLE public.var1 RENAME TO var1_renamed; + +SELECT public.var1_renamed; + +DROP VARIABLE public.var1_renamed; + +SET ROLE TO DEFAULt; + +-- default rights test +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON VARIABLES TO var_test_role; + +CREATE VARIABLE public.var2 AS int; + +SET ROLE TO var_test_role; + +-- should be ok +LET public.var2 = 100; +SELECT public.var2; + +SET ROLE TO DEFAULt; + +DROP VARIABLE public.var2; +DROP OWNED BY var_test_role; + +DROP ROLE var_test_role; + +CREATE VARIABLE xx AS text DEFAULT 'hello'; + +SELECT xx, upper(xx); + +LET xx = 'Hi'; + +SELECT xx; + +DROP VARIABLE xx; + +-- ON TRANSACTION END RESET tests +CREATE VARIABLE t1 AS int DEFAULT -1 ON TRANSACTION END RESET; + +BEGIN; + SELECT t1; + LET t1 = 100; + SELECT t1; +COMMIT; + +SELECT t1; + +BEGIN; + SELECT t1; + LET t1 = 100; + SELECT t1; +ROLLBACK; + +SELECT t1; + +DROP VARIABLE t1; + +CREATE VARIABLE v1 AS int DEFAULT 0; +CREATE VARIABLE v2 AS text DEFAULT 'none'; + +LET v1 = 100; +LET v2 = 'Hello'; +SELECT v1, v2; +LET v1 = DEFAULT; +LET v2 = DEFAULT; +SELECT v1, v2; + +DROP VARIABLE v1; +DROP VARIABLE v2; + +-- ON COMMIT DROP tests +-- should be 0 always +SELECT count(*) FROM pg_variable; + +CREATE TEMP VARIABLE g AS int ON COMMIT DROP; + +SELECT count(*) FROM pg_variable; + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; +COMMIT; + +SELECT count(*) FROM pg_variable; + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; +ROLLBACK; + +SELECT count(*) FROM pg_variable; + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; + LET g = 1; + DISCARD VARIABLES; +COMMIT; + +SELECT count(*) FROM pg_variable; + +BEGIN; + CREATE TEMP VARIABLE g AS int ON COMMIT DROP; + LET g = 1; + DISCARD VARIABLES; +ROLLBACK; + +SELECT count(*) FROM pg_variable; + +-- Encourage use of parallel plans +SET parallel_setup_cost = 0; +SET parallel_tuple_cost = 0; +SET min_parallel_table_scan_size = 0; +SET max_parallel_workers_per_gather = 2; + +-- test on query with workers +CREATE TABLE svar_test(a int); +INSERT INTO svar_test SELECT * FROM generate_series(1,1000); +ANALYZE svar_test; +CREATE VARIABLE zero int; +LET zero = 0; + +-- parallel workers should be used +EXPLAIN (costs off) SELECT count(*) FROM svar_test WHERE a%10 = zero; + +-- result should be 100 +SELECT count(*) FROM svar_test WHERE a%10 = zero; + +LET zero = (SELECT count(*) FROM svar_test); + +-- result should be 1000 +SELECT zero; + +-- parallel workers should be used +EXPLAIN (costs off) LET zero = (SELECT count(*) FROM svar_test); + +-- test for dependency on relation +CREATE VARIABLE v_table AS svar_test; + +ALTER TABLE svar_test ALTER COLUMN a TYPE text; + +DROP VARIABLE v_table; +DROP TABLE svar_test; +DROP VARIABLE zero; + +RESET parallel_setup_cost; +RESET parallel_tuple_cost; +RESET min_parallel_table_scan_size; +RESET max_parallel_workers_per_gather; + +-- use variables in prepared statements +CREATE VARIABLE v AS numeric; +LET v = 3.14; + +-- use variables in views +CREATE VIEW vv AS SELECT COALESCE(v, 0) + 1000 AS result; +SELECT * FROM vv; + +-- start a new session +\c + +SET search_path to svartest; + +SELECT * FROM vv; +LET v = 3.14; +SELECT * FROM vv; + +-- should fail, dependency +DROP VARIABLE v; + +-- should be ok +DROP VARIABLE v CASCADE; + +-- other features +CREATE VARIABLE dt AS integer DEFAULT 0; + +LET dt = 100; +SELECT dt; + +DISCARD VARIABLES; + +SELECT dt; + +DROP VARIABLE dt; + +-- NOT NULL +CREATE VARIABLE v1 AS int NOT NULL; +CREATE VARIABLE v2 AS int NOT NULL DEFAULT NULL; + +-- should fail +SELECT v1; +SELECT v2; +LET v1 = NULL; +LET v2 = NULL; +LET v1 = DEFAULT; +LET v2 = DEFAULT; + +-- should be ok +LET v1 = 100; +LET v2 = 1000; +SELECT v1, v2; + +DROP VARIABLE v1; +DROP VARIABLE v2; + +CREATE VARIABLE tv AS int; +CREATE VARIABLE IF NOT EXISTS tv AS int; +DROP VARIABLE tv; + +CREATE IMMUTABLE VARIABLE iv AS int DEFAULT 100; +SELECT iv; + +-- should fail; +LET iv = 10000; + +DROP VARIABLE iv; + +-- different order +CREATE IMMUTABLE VARIABLE iv AS int DEFAULT 100; +-- should to fail +LET iv = 10000; +-- should be ok +SELECT iv; + +DROP VARIABLE iv; + +CREATE IMMUTABLE VARIABLE iv AS int; + +-- should be ok +LET iv = NULL; + +-- should fail +LET iv = NULL; + +DROP VARIABLE iv; + +-- create variable inside plpgsql block +DO $$ +BEGIN + CREATE VARIABLE do_test_svar AS date DEFAULT '2000-01-01'; +END; +$$; + +SELECT do_test_svar; + +DROP VARIABLE do_test_svar; + +-- should fail +CREATE IMMUTABLE VARIABLE xx AS int NOT NULL; + + + +-- REASSIGN OWNED test +CREATE ROLE var_test_role1; +CREATE ROLE var_test_role2; + +CREATE VARIABLE xxx_var AS int; + +ALTER VARIABLE xxx_var OWNER TO var_test_role1; +REASSIGN OWNED BY var_test_role1 to var_test_role2; + +SELECT varowner::regrole FROM pg_variable WHERE varname = 'xxx_var'; + +DROP OWNED BY var_test_role1; +DROP ROLE var_test_role1; +SELECT count(*) FROM pg_variable WHERE varname = 'xxx_var'; + +DROP OWNED BY var_test_role2; +DROP ROLE var_test_role2; +SELECT count(*) FROM pg_variable WHERE varname = 'xxx_var'; + +-- creating, dropping temporary variable +BEGIN; + +CREATE TEMP VARIABLE tempvar AS INT ON COMMIT DROP; + +LET tempvar = 100; + +SAVEPOINT s1; + +DROP VARIABLE tempvar; + +ROLLBACK TO s1; + +SELECT tempvar; + +COMMIT; + +-- should to fail +LET tempvar = 100; + +BEGIN; + +SAVEPOINT s1; + +CREATE TEMP VARIABLE tempvar AS INT ON COMMIT DROP; + +LET tempvar = 100; + +ROLLBACK TO s1; + +COMMIT; + +-- should to fail +LET tempvar = 100; + +CREATE VARIABLE var1 AS int; +LET var1 = 100; +BEGIN; +DROP VARIABLE var1; +ROLLBACK; +SELECT var1; + +DROP VARIABLE var1; + +CREATE VARIABLE var1 AS int DEFAULT 100; +COMMENT ON VARIABLE var1 IS 'some variable comment'; + +SELECT pg_catalog.obj_description(oid, 'pg_variable') FROM pg_variable WHERE varname = 'var1'; + +DROP VARIABLE var1; + +CREATE TABLE xxtab(avar int); + +CREATE TYPE xxtype AS (avar int); + +CREATE VARIABLE xxtab AS xxtype; + +INSERT INTO xxtab VALUES(10); + +-- it is ambiguous, but columns are preferred +-- variable is shadowed +SELECT xxtab.avar FROM xxtab; + +SET session_variables_ambiguity_warning TO on; + +SELECT xxtab.avar FROM xxtab; + +SET search_path = svartest; + +CREATE VARIABLE testvar as int; + +-- plpgsql variables are preferred against session variables +DO $$ +<<myblock>> +DECLARE testvar int; +BEGIN + -- should be ok without warning + LET testvar = 100; + -- should be ok without warning + testvar := 1000; + -- should be ok without warning + RAISE NOTICE 'session variable is %', svartest.testvar; + -- should be ok without warning + RAISE NOTICE 'plpgsql variable is %', myblock.testvar; + -- should to print plpgsql variable with warning + RAISE NOTICE 'variable is %', testvar; +END; +$$; + +DROP VARIABLE testvar; + +SET session_variables_ambiguity_warning TO default; + +-- should be ok +SELECT avar FROM xxtab; + +CREATE VARIABLE public.avar AS int; + +-- should be ok, see the table +SELECT avar FROM xxtab; + +-- should be ok +SELECT public.avar FROM xxtab; + +DROP VARIABLE xxtab; + +SELECT xxtab.avar FROM xxtab; + +DROP VARIABLE public.avar; + +DROP TYPE xxtype; + +DROP TABLE xxtab; + +-- test of plan cache invalidation +CREATE VARIABLE xx AS int; + +SET plan_cache_mode = force_generic_plan; + +PREPARE pp AS SELECT xx; + +EXECUTE pp; + +DROP VARIABLE xx; + +CREATE VARIABLE xx AS int; + +-- should to work +EXECUTE pp; + +DROP VARIABLE xx; + +DEALLOCATE pp; + +SET plan_cache_mode = DEFAULT; + +CREATE ROLE var_test_role; + +CREATE SCHEMA vartest; + +GRANT USAGE ON SCHEMA vartest TO var_test_role; + +CREATE VARIABLE vartest.x AS int; +CREATE VARIABLE vartest.y AS int; + +LET vartest.x = 100; +LET vartest.y = 101; + +GRANT SELECT ON ALL VARIABLES IN SCHEMA vartest TO var_test_role; + +SET ROLE TO var_test_role; + +SELECT vartest.x, vartest.y; + +SET ROLE TO DEFAULT; + +REVOKE SELECT ON ALL VARIABLES IN SCHEMA vartest FROM var_test_role; + +SET ROLE TO var_test_role; + +-- should to fail +SELECT vartest.x; +SELECT vartest.y; + +SET ROLE TO DEFAULT; + +DROP VARIABLE vartest.x, vartest.y; + +DROP SCHEMA vartest; + +DROP ROLE var_test_role; + +-- test cached plan +CREATE VARIABLE v1 AS text; +CREATE VARIABLE v2 AS int; +CREATE VARIABLE v3 AS int; + +LET v1 = 'test'; +LET v2 = 10; +LET v3 = 5; + +PREPARE q1 AS SELECT v1 || i FROM generate_series(1, v2) g(i) WHERE i IN (v2, v3); + +SET plan_cache_mode to force_generic_plan; + +EXECUTE q1; + +EXPLAIN EXECUTE q1; + +-- dependecy check +DROP VARIABLE v3; + +-- recreate v3 again +CREATE VARIABLE v3 AS int DEFAULT 6; + +-- should to work, the plan should be recreated +EXECUTE q1; + +DEALLOCATE q1; + +-- fill v1 by long text +LET v1 = repeat(' ', 10000); + +PREPARE q1 AS SELECT length(v1); + +EXECUTE q1; + +LET v1 = repeat(' ', 5000); + +EXECUTE q1; + +DEALLOCATE q1; + +SET plan_cache_mode to default; + +DROP VARIABLE v1, v2, v3; + +CREATE ROLE var_test_role; + +CREATE VARIABLE public.v1 AS int DEFAULT 0; + +-- check acl when variable is acessed by simple eval expr method +CREATE OR REPLACE FUNCTION public.fx_var(int) +RETURNS int AS $$ +DECLARE xx int; +BEGIN + xx := public.v1 + $1; + RETURN xx; +END; +$$ LANGUAGE plpgsql; + +-- should be ok +SELECT public.fx_var(0); + +SET ROLE TO var_test_role; + +-- should to fail +SELECT public.fx_var(0); + +SET ROLE TO default; + +GRANT SELECT ON VARIABLE public.v1 TO var_test_role; + +SET ROLE TO var_test_role; + +-- should be ok +SELECT public.fx_var(0); + +SET ROLE TO default; + +REVOKE SELECT ON VARIABLE public.v1 FROM var_test_role; + +SET ROLE TO var_test_role; + +-- should be fail +SELECT public.fx_var(0); + +SET ROLE TO DEFAULT; + +DROP FUNCTION public.fx_var(int); + +DROP VARIABLE public.v1; + +DROP ROLE var_test_role; + +CREATE TYPE public.svar_test_type AS (a int, b int, c numeric); + +CREATE VARIABLE public.svar AS public.svar_test_type; + +LET public.svar = ROW(10,20,30); + +SELECT public.svar; + +ALTER TYPE public.svar_test_type DROP ATTRIBUTE c; + +-- should be ok +SELECT public.svar; + +ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; + +-- should be ok +SELECT public.svar; + +LET public.svar = ROW(10,20,30); + +-- should be ok again for new value +SELECT public.svar; + +DROP VARIABLE public.svar; + +DROP TYPE public.svar_test_type; + +CREATE VARIABLE public.svar AS int; +select oid as varid from pg_variable where varname = 'svar' and varnamespace = 'public'::regnamespace \gset + +SELECT schema, name, removed FROM pg_session_variables(); + +LET public.svar = 100; + +SELECT schema, name, removed FROM pg_session_variables(); + +BEGIN; + +DROP VARIABLE public.svar; + +-- value should be in memory +SELECT schema, removed FROM pg_session_variables(); + +ROLLBACK; + +-- value should be in memory +SELECT schema, name, removed FROM pg_session_variables() f WHERE f.varid = :varid; + +SELECT public.svar; + +BEGIN; + +DROP VARIABLE public.svar; + +-- value should be in memory +SELECT schema, name = :varid::text AS ok, removed FROM pg_session_variables() f WHERE f.varid = :varid;; + +COMMIT; + +-- the memory should be clean; +SELECT schema, name, removed FROM pg_session_variables() WHERE f.varid = :varid; + +BEGIN; + +CREATE VARIABLE public.svar AS int; +select oid as varid from pg_variable where varname = 'svar' and varnamespace = 'public'::regnamespace \gset + + +LET public.svar = 100; + +ROLLBACK; + +-- the memory should be clean; +SELECT schema, name, removed FROM pg_session_variables() f WHERE f.varid = :varid; + +\unset varid + +CREATE VARIABLE public.svar AS int; + +LET public.svar = 100; + +-- repeated aborted transaction +BEGIN; DROP VARIABLE public.svar; ROLLBACK; +BEGIN; DROP VARIABLE public.svar; ROLLBACK; +BEGIN; DROP VARIABLE public.svar; ROLLBACK; + +-- the value should be still available +SELECT public.svar; + +DROP VARIABLE public.svar; + +CREATE TYPE public.svar_test_type AS (a int, b int); + +CREATE VARIABLE public.svar AS public.svar_test_type; + +SELECT public.svar; + +ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; + +SELECT public.svar; + +ALTER TYPE public.svar_test_type DROP ATTRIBUTE b; + +SELECT public.svar; + +DROP VARIABLE public.svar; +DROP TYPE public.svar_test_type; + +CREATE TYPE public.svar_test_type AS (a int, b int); + +CREATE VARIABLE public.svar AS public.svar_test_type; +CREATE VARIABLE public.svar2 AS public.svar_test_type; + +LET public.svar = (10, 20); + +ALTER TYPE public.svar_test_type ADD ATTRIBUTE c int; + +SELECT public.svar; + +LET public.svar2 = (10, 20, 30); + +ALTER TYPE public.svar_test_type DROP ATTRIBUTE b; + +SELECT public.svar; +SELECT public.svar2; + +DROP VARIABLE public.svar; +DROP VARIABLE public.svar2; +DROP TYPE public.svar_test_type; + +-- The composite type cannot be changed when it is used +CREATE TYPE public.svar_type AS (a int, b int); + +CREATE VARIABLE public.svar AS public.svar_type; + +-- should to fail +ALTER TYPE public.svar_type ALTER ATTRIBUTE b TYPE numeric; + +DROP VARIABLE public.svar; + +CREATE TYPE public.svar_type2 AS (a int, b int, c public.svar_type); + +CREATE VARIABLE public.svar AS public.svar_type2; + +-- should to fail +ALTER TYPE public.svar_type ALTER ATTRIBUTE b TYPE numeric; + +DROP VARIABLE public.svar; +DROP TYPE public.svar_type2; +DROP TYPE public.svar_type; + +-- The variable can be shadowed by table or by alias +CREATE TYPE public.svar_type AS (a int, b int, c int); +CREATE VARIABLE public.svar AS public.svar_type; + +CREATE TABLE public.svar (a int, b int); + +INSERT INTO public.svar VALUES(10, 20); + +LET public.svar = (100, 200, 300); + +-- should be ok +-- show table +SELECT * FROM public.svar; +SELECT svar.a FROM public.svar; +SELECT svar.* FROM public.svar; + +-- show variable +SELECT public.svar; +SELECT public.svar.c; +SELECT (public.svar).*; + +-- the variable is shadowed, raise error +SELECT public.svar.c FROM public.svar; + +-- can be fixed by alias +SELECT public.svar.c FROM public.svar x; + +-- again with warnings +SET session_variables_ambiguity_warning TO on; + +SELECT * FROM public.svar; +SELECT svar.a FROM public.svar; +SELECT svar.* FROM public.svar; + +-- show variable +SELECT public.svar; +SELECT public.svar.c; +SELECT (public.svar).*; + +-- the variable is shadowed, raise error +SELECT public.svar.c FROM public.svar; + +-- can be fixed by alias +SELECT public.svar.c FROM public.svar x; + +SET session_variables_ambiguity_warning TO DEFAULT; + +DROP VARIABLE public.svar; +DROP TABLE public.svar; +DROP TYPE public.svar_type; + +-- the LET statement should be disallowed in CTE +CREATE VARIABLE public.svar AS int; +WITH x AS (LET public.svar = 100) SELECT * FROM x; + +-- the LET statement can be prepared +PREPARE pp(int) AS LET public.svar = $1; +EXECUTE pp(10); +SELECT public.svar; +EXECUTE pp(1000); +SELECT public.svar; + +-- test of correct dependency +DROP VARIABLE public.svar; +EXECUTE pp(-1); + +DEALLOCATE pp; + +CREATE VARIABLE public.svar2 AS int; +CREATE VARIABLE public.svar1 AS int DEFAULT 100; + +PREPARE pp(int) AS LET public.svar2 = public.svar1 + $1; + +EXECUTE pp(100); + +SELECT public.svar1, public.svar2; + +LET public.svar1 = 1000; + +EXECUTE pp(100); + +SELECT public.svar1, public.svar2; + +DROP VARIABLE public.svar1; + +EXECUTE pp(100); + +DEALLOCATE pp; + +DROP VARIABLE public.svar2; + +CREATE VARIABLE public.svar AS int; + +-- should be ok +LET public.svar = generate_series(1, 1); + +-- should fail +LET public.svar = generate_series(1, 2); +LET public.svar = generate_series(1, 0); + +DROP VARIABLE public.svar; + +SET search_path TO DEFAULT; + +CREATE TYPE ab AS (a integer, b integer); + +CREATE VARIABLE v_ab AS ab; + +CREATE TABLE v_ab (a integer, b integer); + +SET session_variables_ambiguity_warning = 1; + +-- warning should be raised +SELECT v_ab.a FROM v_ab; + +CREATE SCHEMA v_ab; + +CREATE VARIABLE v_ab.a AS integer; + +-- warning should be raised +SELECT v_ab.a FROM v_ab; + +DROP VARIABLE v_ab; +DROP TABLE v_ab; +DROP TYPE ab; + +CREATE VARIABLE myvar AS int; + +SELECT myvar.blabla; + +DROP VARIABLE myvar; + +-- the result of view should be same in parallel mode too +CREATE VARIABLE v1 AS int; +LET v1 = 10; + +CREATE VIEW v1view AS SELECT COALESCE(v1, 0) AS result; + +SELECT * FROM v1view; + +SET debug_parallel_query TO on; + +SELECT * FROM v1view; + +SET debug_parallel_query TO off; + +DROP VIEW v1view; +DROP VARIABLE v1; + +-- the value should not be corrupted +CREATE VARIABLE v1 text DEFAULT 'abc'; + +CREATE FUNCTION ffunc() +RETURNS text AS $$ +BEGIN + RETURN gfunc(v1); +END +$$ LANGUAGE plpgsql; + +CREATE FUNCTION gfunc(t text) +RETURNS text AS $$ +BEGIN + LET v1 = 'BOOM!'; + RETURN t; +END; +$$ LANGUAGE plpgsql; + +select ffunc(); + +DROP FUNCTION ffunc(); +DROP FUNCTION gfunc(text); + +DROP VARIABLE v1; + +CREATE TYPE t_am_type AS (b int); +CREATE SCHEMA xxx_am; + +SET search_path TO public; + +CREATE VARIABLE xxx_am AS t_am_type DEFAULT row(10); +SELECT xxx_am; +SELECT public.xxx_am; + +CREATE VARIABLE xxx_am.b AS int DEFAULT 20; + +-- the reference should be ambiguous +SELECT xxx_am.b; + +-- enhanced references should be ok +SELECT public.xxx_am.b; +SELECT :"DBNAME".xxx_am.b; + +SET session_variables_ambiguity_warning TO on; + +CREATE TABLE xxx_am(b int); + +-- should be warning, not error +SELECT xxx_am.b FROM xxx_am; + +-- no warning +SELECT x.b FROM xxx_am x; + +DROP TABLE xxx_am; +DROP VARIABLE public.xxx_am; +DROP VARIABLE xxx_am.b; +DROP SCHEMA xxx_am; + +CREATE SCHEMA :"DBNAME"; + +CREATE VARIABLE :"DBNAME".:"DBNAME".:"DBNAME" AS t_am_type DEFAULT row(10); +CREATE VARIABLE :"DBNAME".:"DBNAME".b AS int DEFAULT 20; + +SET search_path to :"DBNAME"; + +-- In this test case, error (and warning) messages contains database name. +-- It is "regression", but it can be possibly different, so we try to +-- supress error and warning messages. + +\set ECHO none +SET client_min_messages TO error; + +-- should be ambiguous +SELECT :"DBNAME".b; + +-- should be true +\echo :ERROR, :SQLSTATE + +-- should be ambiguous too +SELECT :"DBNAME".:"DBNAME".b; + +-- should be true +\echo :ERROR, :SQLSTATE + +CREATE TABLE :"DBNAME"(b int); + +-- should be warning, not error +SELECT :"DBNAME".b FROM :"DBNAME"; + +-- should be false +\echo :ERROR, :SQLSTATE + +\set ECHO all + +DROP TABLE :"DBNAME"; + +DROP VARIABLE :"DBNAME".:"DBNAME".b; +DROP VARIABLE :"DBNAME".:"DBNAME".:"DBNAME"; +DROP SCHEMA :"DBNAME"; + +SET client_min_messages TO DEFAULT; + +SET search_path to public; -- 2.40.1 [text/x-patch] v20230517-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch (27.2K, ../../CAFj8pRAM60yANCnqKHHXZM3j++wpXWOu+rw6tVq5m531LHXiSw@mail.gmail.com/4-v20230517-1-0009-this-patch-changes-error-message-column-doesn-t-exis.patch) download | inline diff: From 431cc6272be696fccfad6d8395f631f8d9a19d68 Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Thu, 24 Nov 2022 12:35:57 +0100 Subject: [PATCH 09/10] this patch changes error message "column doesn't exist" to message "column or variable doesn't exist" The error message will be more correct. Today, missing PL/pgSQL variable can be reported. The change has impact on lot of regress tests not related to session variables, and then it is distributed as separate patch --- src/backend/parser/parse_expr.c | 2 +- src/backend/parser/parse_relation.c | 24 +++++++++++--- src/backend/parser/parse_target.c | 8 +++-- src/include/parser/parse_expr.h | 1 + src/pl/plpgsql/src/expected/plpgsql_array.out | 2 +- .../plpgsql/src/expected/plpgsql_record.out | 4 +-- src/pl/tcl/expected/pltcl_queries.out | 12 +++---- src/test/regress/expected/alter_table.out | 32 +++++++++---------- src/test/regress/expected/copy2.out | 2 +- src/test/regress/expected/errors.out | 8 ++--- src/test/regress/expected/join.out | 12 +++---- src/test/regress/expected/namespace.out | 2 +- src/test/regress/expected/numerology.out | 2 +- src/test/regress/expected/plpgsql.out | 12 +++---- src/test/regress/expected/psql.out | 2 +- src/test/regress/expected/rules.out | 2 +- .../regress/expected/session_variables.out | 2 +- src/test/regress/expected/transactions.out | 4 +-- src/test/regress/expected/union.out | 2 +- 19 files changed, 77 insertions(+), 58 deletions(-) diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 8889ffeb0d..125de5520f 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -486,7 +486,7 @@ transformIndirection(ParseState *pstate, A_Indirection *ind) * When we are in an expression where session variables cannot be used, we raise * the first form of error message. */ -static bool +bool expr_kind_allows_session_variables(ParseExprKind p_expr_kind) { bool result = false; diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c index 41d60494b9..8f3399a42e 100644 --- a/src/backend/parser/parse_relation.c +++ b/src/backend/parser/parse_relation.c @@ -27,6 +27,7 @@ #include "nodes/makefuncs.h" #include "nodes/nodeFuncs.h" #include "parser/parse_enr.h" +#include "parser/parse_expr.h" #include "parser/parse_relation.h" #include "parser/parse_type.h" #include "parser/parsetree.h" @@ -3651,6 +3652,19 @@ errorMissingRTE(ParseState *pstate, RangeVar *relation) parser_errposition(pstate, relation->location))); } +/* + * set message "column does not exist" or "column or variable does not exist" + * in dependency if expression context allows session variables. + */ +static int +column_or_variable_does_not_exists(ParseState *pstate, const char *colname) +{ + if (expr_kind_allows_session_variables(pstate->p_expr_kind)) + return errmsg("column or variable \"%s\" does not exist", colname); + else + return errmsg("column \"%s\" does not exist", colname); +} + /* * Generate a suitable error about a missing column. * @@ -3685,7 +3699,7 @@ errorMissingColumn(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), relname ? errmsg("column %s.%s does not exist", relname, colname) : - errmsg("column \"%s\" does not exist", colname), + column_or_variable_does_not_exists(pstate, colname), errdetail("There are columns named \"%s\", but they are in tables that cannot be referenced from this part of the query.", colname), !relname ? errhint("Try using a table-qualified name.") : 0, @@ -3695,7 +3709,7 @@ errorMissingColumn(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), relname ? errmsg("column %s.%s does not exist", relname, colname) : - errmsg("column \"%s\" does not exist", colname), + column_or_variable_does_not_exists(pstate, colname), errdetail("There is a column named \"%s\" in table \"%s\", but it cannot be referenced from this part of the query.", colname, state->rexact1->eref->aliasname), rte_visible_if_lateral(pstate, state->rexact1) ? @@ -3713,14 +3727,14 @@ errorMissingColumn(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), relname ? errmsg("column %s.%s does not exist", relname, colname) : - errmsg("column \"%s\" does not exist", colname), + column_or_variable_does_not_exists(pstate, colname), parser_errposition(pstate, location))); /* Handle case where we have a single alternative spelling to offer */ ereport(ERROR, (errcode(ERRCODE_UNDEFINED_COLUMN), relname ? errmsg("column %s.%s does not exist", relname, colname) : - errmsg("column \"%s\" does not exist", colname), + column_or_variable_does_not_exists(pstate, colname), errhint("Perhaps you meant to reference the column \"%s.%s\".", state->rfirst->eref->aliasname, strVal(list_nth(state->rfirst->eref->colnames, @@ -3734,7 +3748,7 @@ errorMissingColumn(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), relname ? errmsg("column %s.%s does not exist", relname, colname) : - errmsg("column \"%s\" does not exist", colname), + column_or_variable_does_not_exists(pstate, colname), errhint("Perhaps you meant to reference the column \"%s.%s\" or the column \"%s.%s\".", state->rfirst->eref->aliasname, strVal(list_nth(state->rfirst->eref->colnames, diff --git a/src/backend/parser/parse_target.c b/src/backend/parser/parse_target.c index 022dde27f9..744a7673b3 100644 --- a/src/backend/parser/parse_target.c +++ b/src/backend/parser/parse_target.c @@ -779,7 +779,9 @@ transformAssignmentIndirection(ParseState *pstate, if (!typrelid) ereport(ERROR, (errcode(ERRCODE_DATATYPE_MISMATCH), - errmsg("cannot assign to field \"%s\" of column \"%s\" because its type %s is not a composite type", + errmsg(expr_kind_allows_session_variables(pstate->p_expr_kind) ? + "cannot assign to field \"%s\" of column or variable \"%s\" because its type %s is not a composite type" : + "cannot assign to field \"%s\" of column \"%s\" because its type %s is not a composite type", strVal(n), targetName, format_type_be(targetTypeId)), parser_errposition(pstate, location))); @@ -788,7 +790,9 @@ transformAssignmentIndirection(ParseState *pstate, if (attnum == InvalidAttrNumber) ereport(ERROR, (errcode(ERRCODE_UNDEFINED_COLUMN), - errmsg("cannot assign to field \"%s\" of column \"%s\" because there is no such column in data type %s", + errmsg(expr_kind_allows_session_variables(pstate->p_expr_kind) ? + "cannot assign to field \"%s\" of column or variable \"%s\" because there is no such column in data type %s" : + "cannot assign to field \"%s\" of column \"%s\" because there is no such column in data type %s", strVal(n), targetName, format_type_be(targetTypeId)), parser_errposition(pstate, location))); diff --git a/src/include/parser/parse_expr.h b/src/include/parser/parse_expr.h index f6a5ccb0c1..ebd9ed7cbb 100644 --- a/src/include/parser/parse_expr.h +++ b/src/include/parser/parse_expr.h @@ -22,5 +22,6 @@ extern PGDLLIMPORT bool session_variables_ambiguity_warning; extern Node *transformExpr(ParseState *pstate, Node *expr, ParseExprKind exprKind); extern const char *ParseExprKindName(ParseExprKind exprKind); +extern bool expr_kind_allows_session_variables(ParseExprKind p_expr_kind); #endif /* PARSE_EXPR_H */ diff --git a/src/pl/plpgsql/src/expected/plpgsql_array.out b/src/pl/plpgsql/src/expected/plpgsql_array.out index 9e22e56f00..e131febf3d 100644 --- a/src/pl/plpgsql/src/expected/plpgsql_array.out +++ b/src/pl/plpgsql/src/expected/plpgsql_array.out @@ -41,7 +41,7 @@ NOTICE: a = {"(,11)"}, a[1].i = 11 -- perhaps this ought to work, but for now it doesn't: do $$ declare a complex[]; begin a[1:2].i := array[11,12]; raise notice 'a = %', a; end$$; -ERROR: cannot assign to field "i" of column "a" because its type complex[] is not a composite type +ERROR: cannot assign to field "i" of column or variable "a" because its type complex[] is not a composite type LINE 1: a[1:2].i := array[11,12] ^ QUERY: a[1:2].i := array[11,12] diff --git a/src/pl/plpgsql/src/expected/plpgsql_record.out b/src/pl/plpgsql/src/expected/plpgsql_record.out index afb922df29..f702c7ad54 100644 --- a/src/pl/plpgsql/src/expected/plpgsql_record.out +++ b/src/pl/plpgsql/src/expected/plpgsql_record.out @@ -135,7 +135,7 @@ ERROR: record "c" has no field "x" CONTEXT: PL/pgSQL assignment "c.x.q1 = 1" PL/pgSQL function inline_code_block line 1 at assignment do $$ declare c nested_int8s; begin c.c2.x = 1; end $$; -ERROR: cannot assign to field "x" of column "c" because there is no such column in data type two_int8s +ERROR: cannot assign to field "x" of column or variable "c" because there is no such column in data type two_int8s LINE 1: c.c2.x = 1 ^ QUERY: c.c2.x = 1 @@ -157,7 +157,7 @@ ERROR: record "c" has no field "x" CONTEXT: PL/pgSQL assignment "b.c.x.q1 = 1" PL/pgSQL function inline_code_block line 1 at assignment do $$ <<b>> declare c nested_int8s; begin b.c.c2.x = 1; end $$; -ERROR: cannot assign to field "x" of column "b" because there is no such column in data type two_int8s +ERROR: cannot assign to field "x" of column or variable "b" because there is no such column in data type two_int8s LINE 1: b.c.c2.x = 1 ^ QUERY: b.c.c2.x = 1 diff --git a/src/pl/tcl/expected/pltcl_queries.out b/src/pl/tcl/expected/pltcl_queries.out index 2d922c2333..3b6506d613 100644 --- a/src/pl/tcl/expected/pltcl_queries.out +++ b/src/pl/tcl/expected/pltcl_queries.out @@ -246,12 +246,12 @@ ERROR: type "b" does not exist select tcl_eval('spi_prepare a "b {"'); ERROR: unmatched open brace in list select tcl_error_handling_test($tcl$spi_prepare "select moo" []$tcl$); - tcl_error_handling_test --------------------------------------- - SQLSTATE: 42703 + - condition: undefined_column + - cursor_position: 8 + - message: column "moo" does not exist+ + tcl_error_handling_test +-------------------------------------------------- + SQLSTATE: 42703 + + condition: undefined_column + + cursor_position: 8 + + message: column or variable "moo" does not exist+ statement: select moo (1 row) diff --git a/src/test/regress/expected/alter_table.out b/src/test/regress/expected/alter_table.out index 3b708c7976..def3039c44 100644 --- a/src/test/regress/expected/alter_table.out +++ b/src/test/regress/expected/alter_table.out @@ -1287,19 +1287,19 @@ select * from atacc1; (1 row) select * from atacc1 order by a; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: select * from atacc1 order by a; ^ select * from atacc1 order by "........pg.dropped.1........"; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: select * from atacc1 order by "........pg.dropped.1........"... ^ select * from atacc1 group by a; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: select * from atacc1 group by a; ^ select * from atacc1 group by "........pg.dropped.1........"; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: select * from atacc1 group by "........pg.dropped.1........"... ^ select atacc1.* from atacc1; @@ -1309,7 +1309,7 @@ select atacc1.* from atacc1; (1 row) select a from atacc1; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: select a from atacc1; ^ select atacc1.a from atacc1; @@ -1323,15 +1323,15 @@ select b,c,d from atacc1; (1 row) select a,b,c,d from atacc1; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: select a,b,c,d from atacc1; ^ select * from atacc1 where a = 1; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: select * from atacc1 where a = 1; ^ select "........pg.dropped.1........" from atacc1; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: select "........pg.dropped.1........" from atacc1; ^ select atacc1."........pg.dropped.1........" from atacc1; @@ -1339,11 +1339,11 @@ ERROR: column atacc1.........pg.dropped.1........ does not exist LINE 1: select atacc1."........pg.dropped.1........" from atacc1; ^ select "........pg.dropped.1........",b,c,d from atacc1; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: select "........pg.dropped.1........",b,c,d from atacc1; ^ select * from atacc1 where "........pg.dropped.1........" = 1; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: select * from atacc1 where "........pg.dropped.1........" = ... ^ -- UPDATEs @@ -1352,7 +1352,7 @@ ERROR: column "a" of relation "atacc1" does not exist LINE 1: update atacc1 set a = 3; ^ update atacc1 set b = 2 where a = 3; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: update atacc1 set b = 2 where a = 3; ^ update atacc1 set "........pg.dropped.1........" = 3; @@ -1360,7 +1360,7 @@ ERROR: column "........pg.dropped.1........" of relation "atacc1" does not exis LINE 1: update atacc1 set "........pg.dropped.1........" = 3; ^ update atacc1 set b = 2 where "........pg.dropped.1........" = 3; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: update atacc1 set b = 2 where "........pg.dropped.1........"... ^ -- INSERTs @@ -1408,11 +1408,11 @@ LINE 1: insert into atacc1 ("........pg.dropped.1........",b,c,d) va... ^ -- DELETEs delete from atacc1 where a = 3; -ERROR: column "a" does not exist +ERROR: column or variable "a" does not exist LINE 1: delete from atacc1 where a = 3; ^ delete from atacc1 where "........pg.dropped.1........" = 3; -ERROR: column "........pg.dropped.1........" does not exist +ERROR: column or variable "........pg.dropped.1........" does not exist LINE 1: delete from atacc1 where "........pg.dropped.1........" = 3; ^ delete from atacc1; @@ -1698,7 +1698,7 @@ select f1 from c1; alter table c1 drop column f1; select f1 from c1; -ERROR: column "f1" does not exist +ERROR: column or variable "f1" does not exist LINE 1: select f1 from c1; ^ HINT: Perhaps you meant to reference the column "c1.f2". @@ -1712,7 +1712,7 @@ ERROR: cannot drop inherited column "f1" alter table p1 drop column f1; -- c1.f1 is dropped now, since there is no local definition for it select f1 from c1; -ERROR: column "f1" does not exist +ERROR: column or variable "f1" does not exist LINE 1: select f1 from c1; ^ HINT: Perhaps you meant to reference the column "c1.f2". diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out index 8e33eee719..9b11d87fc6 100644 --- a/src/test/regress/expected/copy2.out +++ b/src/test/regress/expected/copy2.out @@ -122,7 +122,7 @@ LINE 1: COPY x TO stdout WHERE a = 1; COPY x from stdin WHERE a = 50004; COPY x from stdin WHERE a > 60003; COPY x from stdin WHERE f > 60003; -ERROR: column "f" does not exist +ERROR: column or variable "f" does not exist LINE 1: COPY x from stdin WHERE f > 60003; ^ COPY x from stdin WHERE a = max(x.b); diff --git a/src/test/regress/expected/errors.out b/src/test/regress/expected/errors.out index 8c527474da..e53ae451df 100644 --- a/src/test/regress/expected/errors.out +++ b/src/test/regress/expected/errors.out @@ -27,7 +27,7 @@ LINE 1: select * from nonesuch; ^ -- bad name in target list select nonesuch from pg_database; -ERROR: column "nonesuch" does not exist +ERROR: column or variable "nonesuch" does not exist LINE 1: select nonesuch from pg_database; ^ -- empty distinct list isn't OK @@ -37,17 +37,17 @@ LINE 1: select distinct from pg_database; ^ -- bad attribute name on lhs of operator select * from pg_database where nonesuch = pg_database.datname; -ERROR: column "nonesuch" does not exist +ERROR: column or variable "nonesuch" does not exist LINE 1: select * from pg_database where nonesuch = pg_database.datna... ^ -- bad attribute name on rhs of operator select * from pg_database where pg_database.datname = nonesuch; -ERROR: column "nonesuch" does not exist +ERROR: column or variable "nonesuch" does not exist LINE 1: ...ect * from pg_database where pg_database.datname = nonesuch; ^ -- bad attribute name in select distinct on select distinct on (foobar) * from pg_database; -ERROR: column "foobar" does not exist +ERROR: column or variable "foobar" does not exist LINE 1: select distinct on (foobar) * from pg_database; ^ -- grouping with FOR UPDATE diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out index b5f440e43e..51d145edf0 100644 --- a/src/test/regress/expected/join.out +++ b/src/test/regress/expected/join.out @@ -5678,13 +5678,13 @@ LINE 1: select t2.uunique1 from HINT: Perhaps you meant to reference the column "t2.unique1". select uunique1 from tenk1 t1 join tenk2 t2 on t1.two = t2.two; -- error, suggest both at once -ERROR: column "uunique1" does not exist +ERROR: column or variable "uunique1" does not exist LINE 1: select uunique1 from ^ HINT: Perhaps you meant to reference the column "t1.unique1" or the column "t2.unique1". select ctid from tenk1 t1 join tenk2 t2 on t1.two = t2.two; -- error, need qualification -ERROR: column "ctid" does not exist +ERROR: column or variable "ctid" does not exist LINE 1: select ctid from ^ DETAIL: There are columns named "ctid", but they are in tables that cannot be referenced from this part of the query. @@ -6754,7 +6754,7 @@ lateral (select * from int8_tbl t1, -- test some error cases where LATERAL should have been used but wasn't select f1,g from int4_tbl a, (select f1 as g) ss; -ERROR: column "f1" does not exist +ERROR: column or variable "f1" does not exist LINE 1: select f1,g from int4_tbl a, (select f1 as g) ss; ^ DETAIL: There is a column named "f1" in table "a", but it cannot be referenced from this part of the query. @@ -6766,7 +6766,7 @@ LINE 1: select f1,g from int4_tbl a, (select a.f1 as g) ss; DETAIL: There is an entry for table "a", but it cannot be referenced from this part of the query. HINT: To reference that table, you must mark this subquery with LATERAL. select f1,g from int4_tbl a cross join (select f1 as g) ss; -ERROR: column "f1" does not exist +ERROR: column or variable "f1" does not exist LINE 1: select f1,g from int4_tbl a cross join (select f1 as g) ss; ^ DETAIL: There is a column named "f1" in table "a", but it cannot be referenced from this part of the query. @@ -6803,7 +6803,7 @@ LINE 1: select 1 from tenk1 a, lateral (select max(a.unique1) from i... create temp table xx1 as select f1 as x1, -f1 as x2 from int4_tbl; -- error, can't do this: update xx1 set x2 = f1 from (select * from int4_tbl where f1 = x1) ss; -ERROR: column "x1" does not exist +ERROR: column or variable "x1" does not exist LINE 1: ... set x2 = f1 from (select * from int4_tbl where f1 = x1) ss; ^ DETAIL: There is a column named "x1" in table "xx1", but it cannot be referenced from this part of the query. @@ -6823,7 +6823,7 @@ update xx1 set x2 = f1 from xx1, lateral (select * from int4_tbl where f1 = x1) ERROR: table name "xx1" specified more than once -- also errors: delete from xx1 using (select * from int4_tbl where f1 = x1) ss; -ERROR: column "x1" does not exist +ERROR: column or variable "x1" does not exist LINE 1: ...te from xx1 using (select * from int4_tbl where f1 = x1) ss; ^ DETAIL: There is a column named "x1" in table "xx1", but it cannot be referenced from this part of the query. diff --git a/src/test/regress/expected/namespace.out b/src/test/regress/expected/namespace.out index a62fd8ded0..1966b55fea 100644 --- a/src/test/regress/expected/namespace.out +++ b/src/test/regress/expected/namespace.out @@ -23,7 +23,7 @@ BEGIN; SET search_path to public, test_ns_schema_1; CREATE SCHEMA test_ns_schema_2 CREATE VIEW abc_view AS SELECT c FROM abc; -ERROR: column "c" does not exist +ERROR: column or variable "c" does not exist LINE 2: CREATE VIEW abc_view AS SELECT c FROM abc; ^ COMMIT; diff --git a/src/test/regress/expected/numerology.out b/src/test/regress/expected/numerology.out index f662a5050a..fd89def959 100644 --- a/src/test/regress/expected/numerology.out +++ b/src/test/regress/expected/numerology.out @@ -299,7 +299,7 @@ SELECT 1_000.5e0_1; -- error cases SELECT _100; -ERROR: column "_100" does not exist +ERROR: column or variable "_100" does not exist LINE 1: SELECT _100; ^ SELECT 100_; diff --git a/src/test/regress/expected/plpgsql.out b/src/test/regress/expected/plpgsql.out index 272f5d2111..44c3a35461 100644 --- a/src/test/regress/expected/plpgsql.out +++ b/src/test/regress/expected/plpgsql.out @@ -2600,7 +2600,7 @@ end; $$ language plpgsql; -- should fail: SQLSTATE and SQLERRM are only in defined EXCEPTION -- blocks select excpt_test1(); -ERROR: column "sqlstate" does not exist +ERROR: column or variable "sqlstate" does not exist LINE 1: sqlstate ^ QUERY: sqlstate @@ -2615,7 +2615,7 @@ begin end; $$ language plpgsql; -- should fail select excpt_test2(); -ERROR: column "sqlstate" does not exist +ERROR: column or variable "sqlstate" does not exist LINE 1: sqlstate ^ QUERY: sqlstate @@ -4648,7 +4648,7 @@ BEGIN RAISE NOTICE '%, %', r.roomno, r.comment; END LOOP; END$$; -ERROR: column "foo" does not exist +ERROR: column or variable "foo" does not exist LINE 1: SELECT rtrim(roomno) AS roomno, foo FROM Room ORDER BY roomn... ^ QUERY: SELECT rtrim(roomno) AS roomno, foo FROM Room ORDER BY roomno @@ -4690,7 +4690,7 @@ begin raise notice 'x = %', x; end; $$; -ERROR: column "x" does not exist +ERROR: column or variable "x" does not exist LINE 1: x + 1 ^ QUERY: x + 1 @@ -4702,7 +4702,7 @@ begin raise notice 'x = %, y = %', x, y; end; $$; -ERROR: column "x" does not exist +ERROR: column or variable "x" does not exist LINE 1: x + 1 ^ QUERY: x + 1 @@ -5742,7 +5742,7 @@ ALTER TABLE alter_table_under_transition_tables DROP column name; UPDATE alter_table_under_transition_tables SET id = id; -ERROR: column "name" does not exist +ERROR: column or variable "name" does not exist LINE 1: (SELECT string_agg(id || '=' || name, ',') FROM d) ^ QUERY: (SELECT string_agg(id || '=' || name, ',') FROM d) diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out index 956e475447..1fd706a6d0 100644 --- a/src/test/regress/expected/psql.out +++ b/src/test/regress/expected/psql.out @@ -120,7 +120,7 @@ SELECT $1, $2 \bind 'foo' 'bar' \g -- errors -- parse error SELECT foo \bind \g -ERROR: column "foo" does not exist +ERROR: column or variable "foo" does not exist LINE 1: SELECT foo ^ -- tcop error diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 764b827372..22d82333b4 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1191,7 +1191,7 @@ drop rule rules_foorule on rules_foo; -- this should fail because f1 is not exposed for unqualified reference: create rule rules_foorule as on insert to rules_foo where f1 < 100 do instead insert into rules_foo2 values (f1); -ERROR: column "f1" does not exist +ERROR: column or variable "f1" does not exist LINE 2: do instead insert into rules_foo2 values (f1); ^ DETAIL: There are columns named "f1", but they are in tables that cannot be referenced from this part of the query. diff --git a/src/test/regress/expected/session_variables.out b/src/test/regress/expected/session_variables.out index 463b2c96f6..01c234c6f4 100644 --- a/src/test/regress/expected/session_variables.out +++ b/src/test/regress/expected/session_variables.out @@ -273,7 +273,7 @@ SELECT v1; -- should fail LET v1.x = 10; -ERROR: cannot assign to field "x" of column "v1" because there is no such column in data type t1 +ERROR: cannot assign to field "x" of column or variable "v1" because there is no such column in data type t1 LINE 1: LET v1.x = 10; ^ DROP VARIABLE v1; diff --git a/src/test/regress/expected/transactions.out b/src/test/regress/expected/transactions.out index 2b2cff7d91..60c096b857 100644 --- a/src/test/regress/expected/transactions.out +++ b/src/test/regress/expected/transactions.out @@ -256,7 +256,7 @@ SELECT * FROM trans_barbaz; -- should have 1 BEGIN; SAVEPOINT one; SELECT trans_foo; -ERROR: column "trans_foo" does not exist +ERROR: column or variable "trans_foo" does not exist LINE 1: SELECT trans_foo; ^ ROLLBACK TO SAVEPOINT one; @@ -305,7 +305,7 @@ BEGIN; SAVEPOINT one; INSERT INTO savepoints VALUES (5); SELECT trans_foo; -ERROR: column "trans_foo" does not exist +ERROR: column or variable "trans_foo" does not exist LINE 1: SELECT trans_foo; ^ COMMIT; diff --git a/src/test/regress/expected/union.out b/src/test/regress/expected/union.out index e2613d6777..0a6613c430 100644 --- a/src/test/regress/expected/union.out +++ b/src/test/regress/expected/union.out @@ -907,7 +907,7 @@ ORDER BY q2,q1; -- This should fail, because q2 isn't a name of an EXCEPT output column SELECT q1 FROM int8_tbl EXCEPT SELECT q2 FROM int8_tbl ORDER BY q2 LIMIT 1; -ERROR: column "q2" does not exist +ERROR: column or variable "q2" does not exist LINE 1: ... int8_tbl EXCEPT SELECT q2 FROM int8_tbl ORDER BY q2 LIMIT 1... ^ DETAIL: There is a column named "q2" in table "*SELECT* 2", but it cannot be referenced from this part of the query. -- 2.40.1 [text/x-patch] v20230517-1-0006-enhancing-psql-for-session-variables.patch (14.1K, ../../CAFj8pRAM60yANCnqKHHXZM3j++wpXWOu+rw6tVq5m531LHXiSw@mail.gmail.com/5-v20230517-1-0006-enhancing-psql-for-session-variables.patch) download | inline diff: From 6f47286601832c0623d5ee384e15aad2d5b071b0 Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Mon, 4 Apr 2022 22:40:13 +0200 Subject: [PATCH 06/10] enhancing psql for session variables \dV and tab complete for session variables related commands --- src/bin/psql/command.c | 3 ++ src/bin/psql/describe.c | 96 +++++++++++++++++++++++++++++++++++++ src/bin/psql/describe.h | 3 ++ src/bin/psql/help.c | 1 + src/bin/psql/tab-complete.c | 64 +++++++++++++++++++++---- 5 files changed, 157 insertions(+), 10 deletions(-) diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c index 607a57715a..f53244ada3 100644 --- a/src/bin/psql/command.c +++ b/src/bin/psql/command.c @@ -981,6 +981,9 @@ exec_command_d(PsqlScanState scan_state, bool active_branch, const char *cmd) break; } break; + case 'V': /* Variables */ + success = listVariables(pattern, show_verbose); + break; case 'x': /* Extensions */ if (show_verbose) success = listExtensionContents(pattern); diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 50f39a2e50..82a7e20f38 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -5089,6 +5089,102 @@ error_return: return false; } +/* + * \dV + * + * listVariables() + */ +bool +listVariables(const char *pattern, bool verbose) +{ + PQExpBufferData buf; + PGresult *res; + printQueryOpt myopt = pset.popt; + static const bool translate_columns[] = {false, false, false, false, false, false, false, false, false, false, false}; + + initPQExpBuffer(&buf); + + printfPQExpBuffer(&buf, + "SELECT n.nspname as \"%s\",\n" + " v.varname as \"%s\",\n" + " pg_catalog.format_type(v.vartype, v.vartypmod) as \"%s\",\n" + " (SELECT c.collname FROM pg_catalog.pg_collation c, pg_catalog.pg_type bt\n" + " WHERE c.oid = v.varcollation AND bt.oid = v.vartype AND v.varcollation <> bt.typcollation) as \"%s\",\n" + " NOT v.varisnotnull as \"%s\",\n" + " NOT v.varisimmutable as \"%s\",\n" + " pg_catalog.pg_get_expr(v.vardefexpr, 0) as \"%s\",\n" + " pg_catalog.pg_get_userbyid(v.varowner) as \"%s\",\n" + " CASE v.vareoxaction\n" + " WHEN 'd' THEN 'ON COMMIT DROP'\n" + " WHEN 'r' THEN 'ON TRANSACTION END RESET' END as \"%s\"\n", + gettext_noop("Schema"), + gettext_noop("Name"), + gettext_noop("Type"), + gettext_noop("Collation"), + gettext_noop("Nullable"), + gettext_noop("Mutable"), + gettext_noop("Default"), + gettext_noop("Owner"), + gettext_noop("Transactional end action")); + + if (verbose) + { + appendPQExpBufferStr(&buf, ",\n "); + printACLColumn(&buf, "v.varacl"); + appendPQExpBuffer(&buf, + ",\n pg_catalog.obj_description(v.oid, 'pg_variable') AS \"%s\"", + gettext_noop("Description")); + } + + appendPQExpBufferStr(&buf, + "\nFROM pg_catalog.pg_variable v" + "\n LEFT JOIN pg_catalog.pg_namespace n ON n.oid = v.varnamespace"); + + appendPQExpBufferStr(&buf, "\nWHERE true\n"); + if (!pattern) + appendPQExpBufferStr(&buf, " AND n.nspname <> 'pg_catalog'\n" + " AND n.nspname <> 'information_schema'\n"); + + if (!validateSQLNamePattern(&buf, pattern, true, false, + "n.nspname", "v.varname", NULL, + "pg_catalog.pg_variable_is_visible(v.oid)", + NULL, 3)) + return false; + + appendPQExpBufferStr(&buf, "ORDER BY 1,2;"); + + res = PSQLexec(buf.data); + termPQExpBuffer(&buf); + if (!res) + return false; + + /* + * Most functions in this file are content to print an empty table when + * there are no matching objects. We intentionally deviate from that + * here, but only in !quiet mode, for historical reasons. + */ + if (PQntuples(res) == 0 && !pset.quiet) + { + if (pattern) + pg_log_error("Did not find any session variable named \"%s\".", + pattern); + else + pg_log_error("Did not find any session variables."); + } + else + { + myopt.nullPrint = NULL; + myopt.title = _("List of variables"); + myopt.translate_header = true; + myopt.translate_columns = translate_columns; + myopt.n_translate_columns = lengthof(translate_columns); + + printQuery(res, &myopt, pset.queryFout, false, pset.logfile); + } + + PQclear(res); + return true; +} /* * \dFp diff --git a/src/bin/psql/describe.h b/src/bin/psql/describe.h index 554fe86725..55ae238ab2 100644 --- a/src/bin/psql/describe.h +++ b/src/bin/psql/describe.h @@ -146,4 +146,7 @@ extern bool listOpFamilyFunctions(const char *access_method_pattern, /* \dl or \lo_list */ extern bool listLargeObjects(bool verbose); +/* \dV */ +extern bool listVariables(const char *pattern, bool varbose); + #endif /* DESCRIBE_H */ diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index ecfb3c099b..7e2d65c811 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -287,6 +287,7 @@ slashUsage(unsigned short int pager) HELP0(" \\dT[S+] [PATTERN] list data types\n"); HELP0(" \\du[S+] [PATTERN] list roles\n"); HELP0(" \\dv[S+] [PATTERN] list views\n"); + HELP0(" \\dV [PATTERN] list variables\n"); HELP0(" \\dx[+] [PATTERN] list extensions\n"); HELP0(" \\dX [PATTERN] list extended statistics\n"); HELP0(" \\dy[+] [PATTERN] list event triggers\n"); diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index bd04244969..e7caa2ea72 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -962,6 +962,13 @@ static const SchemaQuery Query_for_trigger_of_table = { .refnamespace = "c1.relnamespace", }; +static const SchemaQuery Query_for_list_of_variables = { + .min_server_version = 150000, + .catname = "pg_catalog.pg_variable v", + .viscondition = "pg_catalog.pg_variable_is_visible(v.oid)", + .namespace = "v.varnamespace", + .result = "v.varname", +}; /* * Queries to get lists of names of various kinds of things, possibly @@ -1240,6 +1247,8 @@ static const pgsql_thing_t words_after_create[] = { {"FOREIGN TABLE", NULL, NULL, NULL}, {"FUNCTION", NULL, NULL, Query_for_list_of_functions}, {"GROUP", Query_for_list_of_roles}, + {"IMMUTABLE VARIABLE", NULL, NULL, NULL, NULL, THING_NO_DROP | THING_NO_ALTER}, /* for CREATE IMMUTABLE + * VARIABLE ... */ {"INDEX", NULL, NULL, &Query_for_list_of_indexes}, {"LANGUAGE", Query_for_list_of_languages}, {"LARGE OBJECT", NULL, NULL, NULL, NULL, THING_NO_CREATE | THING_NO_DROP}, @@ -1278,6 +1287,7 @@ static const pgsql_thing_t words_after_create[] = { * TABLE ... */ {"USER", Query_for_list_of_roles, NULL, NULL, Keywords_for_user_thing}, {"USER MAPPING FOR", NULL, NULL, NULL}, + {"VARIABLE", NULL, NULL, &Query_for_list_of_variables}, {"VIEW", NULL, NULL, &Query_for_list_of_views}, {NULL} /* end of list */ }; @@ -1689,8 +1699,8 @@ psql_completion(const char *text, int start, int end) "ABORT", "ALTER", "ANALYZE", "BEGIN", "CALL", "CHECKPOINT", "CLOSE", "CLUSTER", "COMMENT", "COMMIT", "COPY", "CREATE", "DEALLOCATE", "DECLARE", "DELETE FROM", "DISCARD", "DO", "DROP", "END", "EXECUTE", "EXPLAIN", - "FETCH", "GRANT", "IMPORT FOREIGN SCHEMA", "INSERT INTO", "LISTEN", "LOAD", "LOCK", - "MERGE INTO", "MOVE", "NOTIFY", "PREPARE", + "FETCH", "GRANT", "IMPORT FOREIGN SCHEMA", "INSERT INTO", "LET", + "LISTEN", "LOAD", "LOCK", "MERGE INTO", "MOVE", "NOTIFY", "PREPARE", "REASSIGN", "REFRESH MATERIALIZED VIEW", "REINDEX", "RELEASE", "RESET", "REVOKE", "ROLLBACK", "SAVEPOINT", "SECURITY LABEL", "SELECT", "SET", "SHOW", "START", @@ -1710,7 +1720,7 @@ psql_completion(const char *text, int start, int end) "\\dF", "\\dFd", "\\dFp", "\\dFt", "\\dg", "\\di", "\\dl", "\\dL", "\\dm", "\\dn", "\\do", "\\dO", "\\dp", "\\dP", "\\dPi", "\\dPt", "\\drds", "\\dRs", "\\dRp", "\\ds", - "\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy", + "\\dt", "\\dT", "\\dv", "\\du", "\\dx", "\\dX", "\\dy", "\\dV", "\\echo", "\\edit", "\\ef", "\\elif", "\\else", "\\encoding", "\\endif", "\\errverbose", "\\ev", "\\f", @@ -2203,6 +2213,9 @@ psql_completion(const char *text, int start, int end) "ALL"); else if (Matches("ALTER", "SYSTEM", "SET", MatchAny)) COMPLETE_WITH("TO"); + /* ALTER VARIABLE <name> */ + else if (Matches("ALTER", "VARIABLE", MatchAny)) + COMPLETE_WITH("OWNER TO", "RENAME TO", "SET SCHEMA"); /* ALTER VIEW <name> */ else if (Matches("ALTER", "VIEW", MatchAny)) COMPLETE_WITH("ALTER COLUMN", "OWNER TO", "RENAME", @@ -2745,7 +2758,7 @@ psql_completion(const char *text, int start, int end) "ROUTINE", "RULE", "SCHEMA", "SEQUENCE", "SERVER", "STATISTICS", "SUBSCRIPTION", "TABLE", "TABLESPACE", "TEXT SEARCH", "TRANSFORM FOR", - "TRIGGER", "TYPE", "VIEW"); + "TRIGGER", "TYPE", "VARIABLE", "VIEW"); else if (Matches("COMMENT", "ON", "ACCESS", "METHOD")) COMPLETE_WITH_QUERY(Query_for_list_of_access_methods); else if (Matches("COMMENT", "ON", "CONSTRAINT")) @@ -3184,7 +3197,7 @@ psql_completion(const char *text, int start, int end) /* CREATE TABLE --- is allowed inside CREATE SCHEMA, so use TailMatches */ /* Complete "CREATE TEMP/TEMPORARY" with the possible temp objects */ else if (TailMatches("CREATE", "TEMP|TEMPORARY")) - COMPLETE_WITH("SEQUENCE", "TABLE", "VIEW"); + COMPLETE_WITH("IMMUTABLE VARIABLE", "SEQUENCE", "TABLE", "VIEW", "VARIABLE"); /* Complete "CREATE UNLOGGED" with TABLE or MATVIEW */ else if (TailMatches("CREATE", "UNLOGGED")) COMPLETE_WITH("TABLE", "MATERIALIZED VIEW"); @@ -3492,6 +3505,17 @@ psql_completion(const char *text, int start, int end) else if (TailMatches("=", MatchAnyExcept("*)"))) COMPLETE_WITH(",", ")"); } +/* CREATE VARIABLE --- is allowed inside CREATE SCHEMA, so use TailMatches */ + /* Complete CREATE VARIABLE <name> with AS */ + else if (TailMatches("IMMUTABLE")) + COMPLETE_WITH("VARIABLE"); + else if (TailMatches("CREATE", "VARIABLE", MatchAny) || + TailMatches("TEMP|TEMPORARY", "VARIABLE", MatchAny) || + TailMatches("IMMUTABLE", "VARIABLE", MatchAny)) + COMPLETE_WITH("AS"); + else if (TailMatches("VARIABLE", MatchAny, "AS")) + /* Complete CREATE VARIABLE <name> with AS types */ + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_datatypes); /* CREATE VIEW --- is allowed inside CREATE SCHEMA, so use TailMatches */ /* Complete CREATE [ OR REPLACE ] VIEW <name> with AS */ @@ -3607,7 +3631,7 @@ psql_completion(const char *text, int start, int end) /* DISCARD */ else if (Matches("DISCARD")) - COMPLETE_WITH("ALL", "PLANS", "SEQUENCES", "TEMP"); + COMPLETE_WITH("ALL", "PLANS", "SEQUENCES", "TEMP", "VARIABLES"); /* DO */ else if (Matches("DO")) @@ -3734,6 +3758,12 @@ psql_completion(const char *text, int start, int end) else if (Matches("DROP", "TRANSFORM", "FOR", MatchAny, "LANGUAGE", MatchAny)) COMPLETE_WITH("CASCADE", "RESTRICT"); + /* DROP VARIABLE */ + else if (Matches("DROP", "VARIABLE")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_variables); + else if (Matches("DROP", "VARIABLE", MatchAny)) + COMPLETE_WITH("CASCADE", "RESTRICT"); + /* EXECUTE */ else if (Matches("EXECUTE")) COMPLETE_WITH_QUERY(Query_for_list_of_prepared_statements); @@ -3925,7 +3955,7 @@ psql_completion(const char *text, int start, int end) * objects supported. */ if (HeadMatches("ALTER", "DEFAULT", "PRIVILEGES")) - COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS"); + COMPLETE_WITH("TABLES", "SEQUENCES", "FUNCTIONS", "PROCEDURES", "ROUTINES", "TYPES", "SCHEMAS", "VARIABLES"); else COMPLETE_WITH_SCHEMA_QUERY_PLUS(Query_for_list_of_grantables, "ALL FUNCTIONS IN SCHEMA", @@ -3947,7 +3977,8 @@ psql_completion(const char *text, int start, int end) "SEQUENCE", "TABLE", "TABLESPACE", - "TYPE"); + "TYPE", + "VARIABLE"); } else if (TailMatches("GRANT|REVOKE", MatchAny, "ON", "ALL") || TailMatches("REVOKE", "GRANT", "OPTION", "FOR", MatchAny, "ON", "ALL")) @@ -3955,7 +3986,8 @@ psql_completion(const char *text, int start, int end) "PROCEDURES IN SCHEMA", "ROUTINES IN SCHEMA", "SEQUENCES IN SCHEMA", - "TABLES IN SCHEMA"); + "TABLES IN SCHEMA", + "VARIABLES IN SCHEMA"); else if (TailMatches("GRANT|REVOKE", MatchAny, "ON", "FOREIGN") || TailMatches("REVOKE", "GRANT", "OPTION", "FOR", MatchAny, "ON", "FOREIGN")) COMPLETE_WITH("DATA WRAPPER", "SERVER"); @@ -3991,6 +4023,8 @@ psql_completion(const char *text, int start, int end) COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces); else if (TailMatches("TYPE")) COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_datatypes); + else if (TailMatches("VARIABLE")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_variables); else if (TailMatches("GRANT", MatchAny, MatchAny, MatchAny)) COMPLETE_WITH("TO"); else @@ -4275,7 +4309,7 @@ psql_completion(const char *text, int start, int end) /* PREPARE xx AS */ else if (Matches("PREPARE", MatchAny, "AS")) - COMPLETE_WITH("SELECT", "UPDATE", "INSERT INTO", "DELETE FROM"); + COMPLETE_WITH("SELECT", "UPDATE", "INSERT INTO", "DELETE FROM", "LET"); /* * PREPARE TRANSACTION is missing on purpose. It's intended for transaction @@ -4568,6 +4602,14 @@ psql_completion(const char *text, int start, int end) else if (TailMatches("UPDATE", MatchAny, "SET", MatchAnyExcept("*="))) COMPLETE_WITH("="); +/* LET --- can be inside EXPLAIN, PREPARE etc */ + /* If prev. word is LET suggest a list of variables */ + else if (TailMatches("LET")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_variables); + /* Complete LET <variable> with "=" */ + else if (TailMatches("LET", MatchAny)) + COMPLETE_WITH("="); + /* USER MAPPING */ else if (Matches("ALTER|CREATE|DROP", "USER", "MAPPING")) COMPLETE_WITH("FOR"); @@ -4739,6 +4781,8 @@ psql_completion(const char *text, int start, int end) COMPLETE_WITH_QUERY(Query_for_list_of_roles); else if (TailMatchesCS("\\dv*")) COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_views); + else if (TailMatchesCS("\\dV*")) + COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_variables); else if (TailMatchesCS("\\dx*")) COMPLETE_WITH_QUERY(Query_for_list_of_extensions); else if (TailMatchesCS("\\dX*")) -- 2.40.1 [text/x-patch] v20230517-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch (19.6K, ../../CAFj8pRAM60yANCnqKHHXZM3j++wpXWOu+rw6tVq5m531LHXiSw@mail.gmail.com/6-v20230517-1-0007-possibility-to-dump-session-variables-by-pg_dump.patch) download | inline diff: From 4ce93e1bb0c4d3fb365bfcecaaf72f318d1db8ff Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Mon, 4 Apr 2022 20:49:11 +0200 Subject: [PATCH 07/10] possibility to dump session variables by pg_dump Enhancing pg_dump about session variables support --- src/bin/pg_dump/common.c | 3 +- src/bin/pg_dump/dumputils.c | 6 + src/bin/pg_dump/pg_backup.h | 2 + src/bin/pg_dump/pg_backup_archiver.c | 9 + src/bin/pg_dump/pg_dump.c | 237 ++++++++++++++++++++++++++- src/bin/pg_dump/pg_dump.h | 25 ++- src/bin/pg_dump/pg_dump_sort.c | 6 + src/bin/pg_dump/pg_restore.c | 9 +- src/bin/pg_dump/t/002_pg_dump.pl | 82 +++++++++ src/tools/pgindent/typedefs.list | 1 + 10 files changed, 376 insertions(+), 4 deletions(-) diff --git a/src/bin/pg_dump/common.c b/src/bin/pg_dump/common.c index 5d988986ed..4defc8c216 100644 --- a/src/bin/pg_dump/common.c +++ b/src/bin/pg_dump/common.c @@ -264,7 +264,8 @@ getSchemaData(Archive *fout, int *numTablesPtr) pg_log_info("reading subscriptions"); getSubscriptions(fout); - free(inhinfo); /* not needed any longer */ + pg_log_info("reading variables"); + getVariables(fout); *numTablesPtr = numTables; return tblinfo; diff --git a/src/bin/pg_dump/dumputils.c b/src/bin/pg_dump/dumputils.c index 079693585c..bf628834e3 100644 --- a/src/bin/pg_dump/dumputils.c +++ b/src/bin/pg_dump/dumputils.c @@ -511,6 +511,12 @@ do { \ CONVERT_PRIV('r', "SELECT"); CONVERT_PRIV('w', "UPDATE"); } + else if (strcmp(type, "VARIABLE") == 0 || + strcmp(type, "VARIABLES") == 0) + { + CONVERT_PRIV('r', "SELECT"); + CONVERT_PRIV('w', "UPDATE"); + } else abort(); diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index aba780ef4b..a0274bffcb 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -132,12 +132,14 @@ typedef struct _restoreOptions int selFunction; int selTrigger; int selTable; + int selVariable; SimpleStringList indexNames; SimpleStringList functionNames; SimpleStringList schemaNames; SimpleStringList schemaExcludeNames; SimpleStringList triggerNames; SimpleStringList tableNames; + SimpleStringList variableNames; int useDB; ConnParams cparams; /* parameters to use if useDB */ diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index d518349e10..fbbddb6bd9 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -2971,6 +2971,14 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH) !simple_string_list_member(&ropt->triggerNames, te->tag)) return 0; } + else if (strcmp(te->desc, "VARIABLE") == 0) + { + if (!ropt->selVariable) + return 0; + if (ropt->variableNames.head != NULL && + !simple_string_list_member(&ropt->variableNames, te->tag)) + return 0; + } else return 0; } @@ -3459,6 +3467,7 @@ _getObjectDescription(PQExpBuffer buf, const TocEntry *te) strcmp(type, "TEXT SEARCH DICTIONARY") == 0 || strcmp(type, "TEXT SEARCH CONFIGURATION") == 0 || strcmp(type, "TYPE") == 0 || + strcmp(type, "VARIABLE") == 0 || strcmp(type, "VIEW") == 0 || /* non-schema-specified objects */ strcmp(type, "DATABASE") == 0 || diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 41a51ec5cd..dddf741508 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -296,6 +296,7 @@ static void dumpPolicy(Archive *fout, const PolicyInfo *polinfo); static void dumpPublication(Archive *fout, const PublicationInfo *pubinfo); static void dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo); static void dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo); +static void dumpVariable(Archive *fout, const VariableInfo *varinfo); static void dumpDatabase(Archive *fout); static void dumpDatabaseConfig(Archive *AH, PQExpBuffer outbuf, const char *dbname, Oid dboid); @@ -4923,6 +4924,232 @@ get_next_possible_free_pg_type_oid(Archive *fout, PQExpBuffer upgrade_query) return next_possible_free_oid; } +/* + * getVariables + * get information about variables + */ +void +getVariables(Archive *fout) +{ + PQExpBuffer query; + PGresult *res; + VariableInfo *varinfo; + int i_tableoid; + int i_oid; + int i_varname; + int i_varnamespace; + int i_vartype; + int i_vartypname; + int i_vardefexpr; + int i_vareoxaction; + int i_varisnotnull; + int i_varisimmutable; + int i_varowner; + int i_varcollation; + int i_varacl; + int i_acldefault; + int i, + ntups; + + if (fout->remoteVersion < 160000) + return; + + query = createPQExpBuffer(); + + resetPQExpBuffer(query); + + /* Get the variables in current database. */ + appendPQExpBuffer(query, + "SELECT v.tableoid, v.oid, v.varname,\n" + "v.vareoxaction,\n" + "v.varnamespace,\n" + "v.vartype,\n" + "pg_catalog.format_type(v.vartype, v.vartypmod) as vartypname,\n" + "v.varisnotnull,\n" + "v.varisimmutable,\n" + "CASE WHEN v.varcollation <> t.typcollation " + "THEN v.varcollation ELSE 0 END AS varcollation,\n" + "pg_catalog.pg_get_expr(v.vardefexpr,0) as vardefexpr,\n" + "v.varowner,\n" + "v.varacl,\n" + "acldefault('V', v.varowner) AS acldefault\n" + "FROM pg_catalog.pg_variable v\n" + "JOIN pg_catalog.pg_type t " + "ON (v.vartype = t.oid)"); + + res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK); + + ntups = PQntuples(res); + + i_tableoid = PQfnumber(res, "tableoid"); + i_oid = PQfnumber(res, "oid"); + i_varname = PQfnumber(res, "varname"); + i_varnamespace = PQfnumber(res, "varnamespace"); + i_vartype = PQfnumber(res, "vartype"); + i_vartypname = PQfnumber(res, "vartypname"); + i_vareoxaction = PQfnumber(res, "vareoxaction"); + i_vardefexpr = PQfnumber(res, "vardefexpr"); + i_varisnotnull = PQfnumber(res, "varisnotnull"); + i_varisimmutable = PQfnumber(res, "varisimmutable"); + i_varcollation = PQfnumber(res, "varcollation"); + + i_varowner = PQfnumber(res, "varowner"); + i_varacl = PQfnumber(res, "varacl"); + i_acldefault = PQfnumber(res, "acldefault"); + + varinfo = pg_malloc(ntups * sizeof(VariableInfo)); + + for (i = 0; i < ntups; i++) + { + TypeInfo *vtype; + + varinfo[i].dobj.objType = DO_VARIABLE; + varinfo[i].dobj.catId.tableoid = + atooid(PQgetvalue(res, i, i_tableoid)); + varinfo[i].dobj.catId.oid = atooid(PQgetvalue(res, i, i_oid)); + AssignDumpId(&varinfo[i].dobj); + varinfo[i].dobj.name = pg_strdup(PQgetvalue(res, i, i_varname)); + varinfo[i].dobj.namespace = + findNamespace(atooid(PQgetvalue(res, i, i_varnamespace))); + + varinfo[i].vartype = atooid(PQgetvalue(res, i, i_vartype)); + varinfo[i].vartypname = pg_strdup(PQgetvalue(res, i, i_vartypname)); + varinfo[i].vareoxaction = pg_strdup(PQgetvalue(res, i, i_vareoxaction)); + varinfo[i].varisnotnull = *(PQgetvalue(res, i, i_varisnotnull)) == 't'; + varinfo[i].varisimmutable = *(PQgetvalue(res, i, i_varisimmutable)) == 't'; + varinfo[i].varcollation = atooid(PQgetvalue(res, i, i_varcollation)); + + varinfo[i].dacl.acl = pg_strdup(PQgetvalue(res, i, i_varacl)); + varinfo[i].dacl.acldefault = pg_strdup(PQgetvalue(res, i, i_acldefault)); + varinfo[i].dacl.privtype = 0; + varinfo[i].dacl.initprivs = NULL; + varinfo[i].rolname = getRoleName(PQgetvalue(res, i, i_varowner)); + + /* Decide whether we want to dump it */ + selectDumpableObject(&(varinfo[i].dobj), fout); + + /* Do not try to dump ACL if no ACL exists. */ + if (!PQgetisnull(res, i, i_varacl)) + varinfo[i].dobj.components |= DUMP_COMPONENT_ACL; + + if (PQgetisnull(res, i, i_vardefexpr)) + varinfo[i].vardefexpr = NULL; + else + varinfo[i].vardefexpr = pg_strdup(PQgetvalue(res, i, i_vardefexpr)); + + if (strlen(varinfo[i].rolname) == 0) + pg_log_warning("owner of variable \"%s\" appears to be invalid", + varinfo[i].dobj.name); + + /* Decide whether we want to dump it */ + selectDumpableObject(&(varinfo[i].dobj), fout); + + vtype = findTypeByOid(varinfo[i].vartype); + addObjectDependency(&varinfo[i].dobj, vtype->dobj.dumpId); + } + PQclear(res); + + destroyPQExpBuffer(query); +} + +/* + * dumpVariable + * dump the definition of the given session variable + */ +static void +dumpVariable(Archive *fout, const VariableInfo *varinfo) +{ + DumpOptions *dopt = fout->dopt; + + PQExpBuffer delq; + PQExpBuffer query; + char *qualvarname; + const char *vartypname; + const char *vardefexpr; + const char *vareoxaction; + const char *varisimmutable; + Oid varcollation; + bool varisnotnull; + + /* Skip if not to be dumped */ + if (!varinfo->dobj.dump || dopt->dataOnly) + return; + + delq = createPQExpBuffer(); + query = createPQExpBuffer(); + + qualvarname = pg_strdup(fmtQualifiedDumpable(varinfo)); + vartypname = varinfo->vartypname; + vardefexpr = varinfo->vardefexpr; + vareoxaction = varinfo->vareoxaction; + varisnotnull = varinfo->varisnotnull; + varisimmutable = varinfo->varisimmutable ? "IMMUTABLE " : ""; + varcollation = varinfo->varcollation; + + appendPQExpBuffer(delq, "DROP VARIABLE %s;\n", + qualvarname); + + appendPQExpBuffer(query, "CREATE %sVARIABLE %s AS %s", + varisimmutable, qualvarname, vartypname); + + if (OidIsValid(varcollation)) + { + CollInfo *coll; + + coll = findCollationByOid(varcollation); + if (coll) + appendPQExpBuffer(query, " COLLATE %s", + fmtQualifiedDumpable(coll)); + } + + if (varisnotnull) + appendPQExpBuffer(query, " NOT NULL"); + + if (vardefexpr) + appendPQExpBuffer(query, " DEFAULT %s", + vardefexpr); + + if (strcmp(vareoxaction, "d") == 0) + appendPQExpBuffer(query, " ON COMMIT DROP"); + else if (strcmp(vareoxaction, "r") == 0) + appendPQExpBuffer(query, " ON TRANSACTION END RESET"); + + appendPQExpBuffer(query, ";\n"); + + if (varinfo->dobj.dump & DUMP_COMPONENT_DEFINITION) + ArchiveEntry(fout, varinfo->dobj.catId, varinfo->dobj.dumpId, + ARCHIVE_OPTS(.tag = varinfo->dobj.name, + .namespace = varinfo->dobj.namespace->dobj.name, + .owner = varinfo->rolname, + .description = "VARIABLE", + .section = SECTION_PRE_DATA, + .createStmt = query->data, + .dropStmt = delq->data)); + + /* Dump comment if any */ + if (varinfo->dobj.dump & DUMP_COMPONENT_COMMENT) + dumpComment(fout, "VARIABLE", qualvarname, + NULL, varinfo->rolname, + varinfo->dobj.catId, 0, varinfo->dobj.dumpId); + + /* Dump ACL if any */ + if (varinfo->dobj.dump & DUMP_COMPONENT_ACL) + { + char *qvarname = pg_strdup(fmtId(varinfo->dobj.name)); + + dumpACL(fout, varinfo->dobj.dumpId, InvalidDumpId, "VARIABLE", + qvarname, NULL, + varinfo->dobj.namespace->dobj.name, varinfo->rolname, &varinfo->dacl); + + free(qvarname); + } + + destroyPQExpBuffer(delq); + destroyPQExpBuffer(query); + + free(qualvarname); +} + static void binary_upgrade_set_type_oids_by_type_oid(Archive *fout, PQExpBuffer upgrade_buffer, @@ -9662,7 +9889,8 @@ getAdditionalACLs(Archive *fout) dobj->objType == DO_TABLE || dobj->objType == DO_PROCLANG || dobj->objType == DO_FDW || - dobj->objType == DO_FOREIGN_SERVER) + dobj->objType == DO_FOREIGN_SERVER || + dobj->objType == DO_VARIABLE) { DumpableObjectWithAcl *daobj = (DumpableObjectWithAcl *) dobj; @@ -10252,6 +10480,9 @@ dumpDumpableObject(Archive *fout, DumpableObject *dobj) case DO_SUBSCRIPTION: dumpSubscription(fout, (const SubscriptionInfo *) dobj); break; + case DO_VARIABLE: + dumpVariable(fout, (VariableInfo *) dobj); + break; case DO_PRE_DATA_BOUNDARY: case DO_POST_DATA_BOUNDARY: /* never dumped, nothing to do */ @@ -14649,6 +14880,9 @@ dumpDefaultACL(Archive *fout, const DefaultACLInfo *daclinfo) case DEFACLOBJ_NAMESPACE: type = "SCHEMAS"; break; + case DEFACLOBJ_VARIABLE: + type = "VARIABLES"; + break; default: /* shouldn't get here */ pg_fatal("unrecognized object type in default privileges: %d", @@ -18193,6 +18427,7 @@ addBoundaryDependencies(DumpableObject **dobjs, int numObjs, case DO_CONVERSION: case DO_TABLE: case DO_TABLE_ATTACH: + case DO_VARIABLE: case DO_ATTRDEF: case DO_PROCLANG: case DO_CAST: diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index ed6ce41ad7..ab9d2215fb 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -52,6 +52,7 @@ typedef enum DO_TABLE, DO_TABLE_ATTACH, DO_ATTRDEF, + DO_VARIABLE, DO_INDEX, DO_INDEX_ATTACH, DO_STATSEXT, @@ -82,7 +83,7 @@ typedef enum DO_PUBLICATION, DO_PUBLICATION_REL, DO_PUBLICATION_TABLE_IN_SCHEMA, - DO_SUBSCRIPTION + DO_SUBSCRIPTION, } DumpableObjectType; /* @@ -666,6 +667,27 @@ typedef struct _SubscriptionInfo char *subpasswordrequired; } SubscriptionInfo; +/* + * The VariableInfo struct is used to represent session variables + */ +typedef struct _VariableInfo +{ + DumpableObject dobj; + DumpableAcl dacl; + Oid vartype; + char *vartypname; + char *vareoxaction; + char *vardefexpr; + char *varacl; + char *rvaracl; + char *initvaracl; + char *initrvaracl; + bool varisnotnull; + bool varisimmutable; + Oid varcollation; + const char *rolname; /* name of owner, or empty string */ +} VariableInfo; + /* * common utility functions */ @@ -749,5 +771,6 @@ extern void getPublicationNamespaces(Archive *fout); extern void getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables); extern void getSubscriptions(Archive *fout); +extern void getVariables(Archive *fout); #endif /* PG_DUMP_H */ diff --git a/src/bin/pg_dump/pg_dump_sort.c b/src/bin/pg_dump/pg_dump_sort.c index 745578d855..b1dcf98e64 100644 --- a/src/bin/pg_dump/pg_dump_sort.c +++ b/src/bin/pg_dump/pg_dump_sort.c @@ -76,6 +76,7 @@ enum dbObjectTypePriorities PRIO_DUMMY_TYPE, PRIO_ATTRDEF, PRIO_LARGE_OBJECT, + PRIO_VARIABLE, PRIO_PRE_DATA_BOUNDARY, /* boundary! */ PRIO_TABLE_DATA, PRIO_SEQUENCE_SET, @@ -116,6 +117,7 @@ static const int dbObjectTypePriority[] = PRIO_TABLE, /* DO_TABLE */ PRIO_TABLE_ATTACH, /* DO_TABLE_ATTACH */ PRIO_ATTRDEF, /* DO_ATTRDEF */ + PRIO_VARIABLE, /* DO_VARIABLE */ PRIO_INDEX, /* DO_INDEX */ PRIO_INDEX_ATTACH, /* DO_INDEX_ATTACH */ PRIO_STATSEXT, /* DO_STATSEXT */ @@ -1508,6 +1510,10 @@ describeDumpableObject(DumpableObject *obj, char *buf, int bufsize) "POST-DATA BOUNDARY (ID %d)", obj->dumpId); return; + case DO_VARIABLE: + snprintf(buf, bufsize, + "VARIABLE %s (ID %d OID %u)", + obj->name, obj->dumpId, obj->catId.oid); } /* shouldn't get here */ snprintf(buf, bufsize, diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c index 049a100634..830cde5421 100644 --- a/src/bin/pg_dump/pg_restore.c +++ b/src/bin/pg_dump/pg_restore.c @@ -103,6 +103,7 @@ main(int argc, char **argv) {"trigger", 1, NULL, 'T'}, {"use-list", 1, NULL, 'L'}, {"username", 1, NULL, 'U'}, + {"variable", 1, NULL, 'A'}, {"verbose", 0, NULL, 'v'}, {"single-transaction", 0, NULL, '1'}, @@ -151,7 +152,7 @@ main(int argc, char **argv) } } - while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1", + while ((c = getopt_long(argc, argv, "A:acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1", cmdopts, NULL)) != -1) { switch (c) @@ -159,6 +160,11 @@ main(int argc, char **argv) case 'a': /* Dump data only */ opts->dataOnly = 1; break; + case 'A': /* vAriable */ + opts->selTypes = 1; + opts->selVariable = 1; + simple_string_list_append(&opts->variableNames, optarg); + break; case 'c': /* clean (i.e., drop) schema prior to create */ opts->dropSchema = 1; break; @@ -444,6 +450,7 @@ usage(const char *progname) printf(_("\nOptions controlling the restore:\n")); printf(_(" -a, --data-only restore only the data, no schema\n")); + printf(_(" -A, --variable=NAME restore named session variable\n")); printf(_(" -c, --clean clean (drop) database objects before recreating\n")); printf(_(" -C, --create create the target database\n")); printf(_(" -e, --exit-on-error exit on error, default is to continue\n")); diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 93e24d5145..f60a117995 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -805,6 +805,16 @@ my %tests = ( unlike => { no_privs => 1, }, }, + 'ALTER DEFAULT PRIVILEGES FOR ROLE regress_dump_test_role GRANT SELECT ON VARIABLES TO PUBLIC' + => { + create_order => 56, + create_sql => 'ALTER DEFAULT PRIVILEGES FOR ROLE regress_dump_test_role GRANT SELECT ON VARIABLES TO PUBLIC;', + regexp => qr/^ + \QALTER DEFAULT PRIVILEGES FOR ROLE regress_dump_test_role GRANT SELECT ON VARIABLES TO PUBLIC;\E/xm, + like => { %full_runs, section_post_data => 1, }, + unlike => { no_privs => 1, }, + }, + 'ALTER ROLE regress_dump_test_role' => { regexp => qr/^ \QALTER ROLE regress_dump_test_role WITH \E @@ -1658,6 +1668,23 @@ my %tests = ( }, }, + 'COMMENT ON VARIABLE dump_test.variable1' => { + create_order => 71, + create_sql => 'COMMENT ON VARIABLE dump_test.variable1 + IS \'comment on variable\';', + regexp => + qr/^\QCOMMENT ON VARIABLE dump_test.variable1 IS 'comment on variable';\E/m, + like => { + %full_runs, + %dump_test_schema_runs, + section_pre_data => 1, + }, + unlike => { + exclude_dump_test_schema => 1, + only_dump_measurement => 1, + }, + }, + 'COPY test_table' => { create_order => 4, create_sql => 'INSERT INTO dump_test.test_table (col1) ' @@ -3852,6 +3879,42 @@ my %tests = ( }, }, + 'CREATE VARIABLE test_variable' => { + all_runs => 1, + catch_all => 'CREATE ... commands', + create_order => 61, + create_sql => 'CREATE VARIABLE dump_test.variable1 AS integer DEFAULT 0;', + regexp => qr/^ + \QCREATE VARIABLE dump_test.variable1 AS integer DEFAULT 0;\E/xm, + like => { + %full_runs, + %dump_test_schema_runs, + section_pre_data => 1, + }, + unlike => { + exclude_dump_test_schema => 1, + only_dump_measurement => 1, + }, + }, + + 'CREATE IMMUTABLE VARIABLE test_variable' => { + all_runs => 1, + catch_all => 'CREATE ... commands', + create_order => 61, + create_sql => 'CREATE IMMUTABLE VARIABLE dump_test.variable2 AS integer DEFAULT 0;', + regexp => qr/^ + \QCREATE IMMUTABLE VARIABLE dump_test.variable2 AS integer DEFAULT 0;\E/xm, + like => { + %full_runs, + %dump_test_schema_runs, + section_pre_data => 1, + }, + unlike => { + exclude_dump_test_schema => 1, + only_dump_measurement => 1, + }, + }, + 'CREATE VIEW test_view' => { create_order => 61, create_sql => 'CREATE VIEW dump_test.test_view @@ -4312,6 +4375,25 @@ my %tests = ( like => {}, }, + 'GRANT SELECT ON VARIABLE dump_test.variable1' => { + create_order => 73, + create_sql => + 'GRANT SELECT ON VARIABLE dump_test.variable1 TO regress_dump_test_role;', + regexp => qr/^ + \QGRANT SELECT ON VARIABLE dump_test.variable1 TO regress_dump_test_role;\E + /xm, + like => { + %full_runs, + %dump_test_schema_runs, + section_pre_data => 1, + }, + unlike => { + exclude_dump_test_schema => 1, + no_privs => 1, + only_dump_measurement => 1, + }, + }, + 'REFRESH MATERIALIZED VIEW matview' => { regexp => qr/^\QREFRESH MATERIALIZED VIEW dump_test.matview;\E/m, like => diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 429fb8cdcf..fac862ca38 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2944,6 +2944,7 @@ Variable VariableAssignHook VariableCache VariableCacheData +VariableInfo VariableSetKind VariableSetStmt VariableShowStmt -- 2.40.1 [text/x-patch] v20230517-1-0010-documentation.patch (46.1K, ../../CAFj8pRAM60yANCnqKHHXZM3j++wpXWOu+rw6tVq5m531LHXiSw@mail.gmail.com/7-v20230517-1-0010-documentation.patch) download | inline diff: From 9cb83c9f3696e01bb0e9a30435665d5c8a0cf6c1 Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Mon, 4 Apr 2022 20:23:54 +0200 Subject: [PATCH 10/10] documentation Documentation for CREATE VARIABLE, DROP VARIABLE and LET commands. Update of GRANT, REVOKE, DISCARD, ALTER commands related to support of session variables. --- doc/src/sgml/catalogs.sgml | 170 +++++++++++++ doc/src/sgml/config.sgml | 60 +++++ doc/src/sgml/ddl.sgml | 64 +++++ doc/src/sgml/event-trigger.sgml | 24 ++ doc/src/sgml/glossary.sgml | 16 ++ doc/src/sgml/plpgsql.sgml | 12 + doc/src/sgml/ref/allfiles.sgml | 4 + .../sgml/ref/alter_default_privileges.sgml | 26 +- doc/src/sgml/ref/alter_variable.sgml | 179 ++++++++++++++ doc/src/sgml/ref/comment.sgml | 1 + doc/src/sgml/ref/create_schema.sgml | 7 +- doc/src/sgml/ref/create_variable.sgml | 223 ++++++++++++++++++ doc/src/sgml/ref/discard.sgml | 14 +- doc/src/sgml/ref/drop_variable.sgml | 118 +++++++++ doc/src/sgml/ref/grant.sgml | 6 + doc/src/sgml/ref/let.sgml | 109 +++++++++ doc/src/sgml/ref/pg_restore.sgml | 11 + doc/src/sgml/ref/revoke.sgml | 7 + doc/src/sgml/reference.sgml | 4 + 19 files changed, 1043 insertions(+), 12 deletions(-) create mode 100644 doc/src/sgml/ref/alter_variable.sgml create mode 100644 doc/src/sgml/ref/create_variable.sgml create mode 100644 doc/src/sgml/ref/drop_variable.sgml create mode 100644 doc/src/sgml/ref/let.sgml diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 5240840552..45989060f2 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -369,6 +369,11 @@ <entry><link linkend="catalog-pg-user-mapping"><structname>pg_user_mapping</structname></link></entry> <entry>mappings of users to foreign servers</entry> </row> + + <row> + <entry><link linkend="catalog-pg-variable"><structname>pg_variable</structname></link></entry> + <entry>session variables</entry> + </row> </tbody> </tgroup> </table> @@ -9677,4 +9682,169 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </table> </sect1> + <sect1 id="catalog-pg-variable"> + <title><structname>pg_variable</structname></title> + + <indexterm zone="catalog-pg-variable"> + <primary>pg_variable</primary> + </indexterm> + + <para> + The table <structname>pg_variable</structname> provides information about + session variables. + </para> + + <table> + <title><structname>pg_variable</structname> Columns</title> + <tgroup cols="1"> + <thead> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + Column Type + </para> + <para> + Description + </para></entry> + </row> + </thead> + + <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>oid</structfield> <type>oid</type> + </para> + <para> + Row identifier + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>vartype</structfield> <type>oid</type> + (references <link linkend="catalog-pg-type"><structname>pg_type</structname></link>.<structfield>oid</structfield>) + </para> + <para> + The OID of the variable's data type + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varcreate_lsn</structfield> <type>XLogRecPtr</type> + </para> + <para> + LSN of the transaction where variable was created. It is used + (in combination with <structfield>oid</structfield>) as everytime + unique identifier. Only <structfield>oid</structfield> cannot be + used for this purpose, because unused <structfield>oid</structfield> + can be reused. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varname</structfield> <type>name</type> + </para> + <para> + Name of the session variable + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varnamespace</structfield> <type>oid</type> + (references <link linkend="catalog-pg-namespace"><structname>pg_namespace</structname></link>.<structfield>oid</structfield>) + </para> + <para> + The OID of the namespace that contains this variable + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varowner</structfield> <type>oid</type> + (references <link linkend="catalog-pg-authid"><structname>pg_authid</structname></link>.<structfield>oid</structfield>) + </para> + <para> + Owner of the variable + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>vartypmod</structfield> <type>int4</type> + </para> + <para> + <structfield>vartypmod</structfield> records type-specific data + supplied at variable creation time (for example, the maximum + length of a <type>varchar</type> column). It is passed to + type-specific input functions and length coercion functions. + The value will generally be -1 for types that do not need <structfield>vartypmod</structfield>. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varisnotnull</structfield> <type>boolean</type> + </para> + <para> + True if the session variable doesn't allow null value. The default value is false. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varisimmutable</structfield> <type>boolean</type> + </para> + <para> + True if the variable is <link linkend="sql-createvariable-immutable">immutable</link> (cannot be modified). The default value is false. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>vareoxaction</structfield> <type>char</type> + </para> + <para> + Action performed at end of transaction: + <literal>n</literal> = no action, <literal>d</literal> = drop the variable, + <literal>r</literal> = reset the variable to its default value. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varcollation</structfield> <type>oid</type> + (references <link linkend="catalog-pg-collation"><structname>pg_collation</structname></link>.<structfield>oid</structfield>) + </para> + <para> + The defined collation of the variable, or zero if the variable is + not of a collatable data type. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>varacl</structfield> <type>aclitem[]</type> + </para> + <para> + Access privileges; see + <xref linkend="sql-grant"/> and + <xref linkend="sql-revoke"/> + for details + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>vardefexpr</structfield> <type>pg_node_tree</type> + </para> + <para> + The internal representation of the variable default value + </para></entry> + </row> + </tbody> + </tgroup> + </table> + </sect1> </chapter> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 2073bafa1f..5a70621082 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -10443,6 +10443,66 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir' </listitem> </varlistentry> + <varlistentry id="guc-session-variables-ambiguity-warning" xreflabel="session_variables_ambiguity_warning"> + <term><varname>session_variables_ambiguity_warning</varname> (<type>boolean</type>) + <indexterm> + <primary><varname>session_variables_ambiguity_warning</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + When on, a warning is raised when any identifier in a query could be + used as both a column identifier, routine variable or a session + variable identifier. The default is <literal>off</literal>. + </para> + <para> + Session variables can be shadowed by column references in a query, this + is an expected behavior. Previously working queries shouldn't error out + by creating any session variable, so session variables are always shadowed + if an identifier is ambiguous. Variables should be referenced using + anunambiguous identifier without any possibility for a collision with + identifier of other database objects (column names or record fields names). + The warning messages emitted when enabling <varname>session_variables_ambiguity_warning</varname> + can help finding such identifier collision. +<programlisting> +CREATE TABLE foo(a int); +INSERT INTO foo VALUES(10); +CREATE VARIABLE a int; +LET a = 100; +SELECT a FROM foo; +</programlisting> + +<screen> + a +---- + 10 +(1 row) +</screen> + +<programlisting> +SET session_variables_ambiguity_warning TO on; +SELECT a FROM foo; +</programlisting> + +<screen> +WARNING: session variable "a" is shadowed +LINE 1: SELECT a FROM foo; + ^ +DETAIL: Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name. + a +---- + 10 +(1 row) +</screen> + </para> + <para> + This feature can significantly increase log size, so it's disabled by + default. For testing or development environments it's recommended to + enable it if you use session variables. + </para> + </listitem> + </varlistentry> + <varlistentry id="guc-standard-conforming-strings" xreflabel="standard_conforming_strings"> <term><varname>standard_conforming_strings</varname> (<type>boolean</type>) <indexterm><primary>strings</primary><secondary>standard conforming</secondary></indexterm> diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index 458b89c0cb..e953c588f3 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -5112,6 +5112,70 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01'; </para> </sect1> + <sect1 id="ddl-session-variables"> + <title>Session Variables</title> + + <indexterm zone="ddl-session-variables"> + <primary>Session variables</primary> + </indexterm> + + <indexterm> + <primary>session variable</primary> + </indexterm> + + <para> + Session variables are database objects that can hold a value. + Session variables, like relations, exist within a schema and their access + is controlled via <command>GRANT</command> and <command>REVOKE</command> + commands. A session variable can be created by the <command>CREATE + VARIABLE</command> command. + </para> + + <para> + The value of a session variable is set with the <command>LET</command> SQL + command. While session variables share properties with tables, their value + cannot be updated with an <command>UPDATE</command> command. The value of a + session variable may be retrieved by the <command>SELECT</command> SQL + command. +<programlisting> +CREATE VARIABLE var1 AS date; +LET var1 = current_date; +SELECT var1; +</programlisting> + + or + +<programlisting> +CREATE VARIABLE public.current_user_id AS integer; +GRANT READ ON VARIABLE public.current_user_id TO PUBLIC; +LET current_user_id = (SELECT id FROM users WHERE usename = session_user); +SELECT current_user_id; +</programlisting> + </para> + + <para> + The value of a session variable is local to the current session. Retrieving + a variable's value returns either a <literal>NULL</literal> or a default + value, unless its value has been set to something else in the current + session using the <command>LET</command> command. The content of a variable + is not transactional. This is the same as regular variables in PL languages. + The session variables can be persistent or can be temporary. In both cases, + the content of session variables is temporary and not shared (like an + content of temporary tables). + </para> + + <para> + The session variables can be shadowed by column references in a query. When + a query contains identifiers or qualified identifiers that could be used as + both a session variable identifiers and as column identifier, then the + column identifier is preferred every time. Warnings can be emitted when + this situation happens by enabling configuration parameter <xref + linkend="guc-session-variables-ambiguity-warning"/>. User can explicitly + qualify the source object by syntax <literal>table.column</literal> or + <literal>variable.column</literal>. + </para> + </sect1> + <sect1 id="ddl-others"> <title>Other Database Objects</title> diff --git a/doc/src/sgml/event-trigger.sgml b/doc/src/sgml/event-trigger.sgml index 3b6a5361b3..cac5f9ff94 100644 --- a/doc/src/sgml/event-trigger.sgml +++ b/doc/src/sgml/event-trigger.sgml @@ -405,6 +405,14 @@ <entry align="center"><literal>-</literal></entry> <entry align="left"></entry> </row> + <row> + <entry align="left"><literal>ALTER VARIABLE</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>-</literal></entry> + <entry align="center"><literal>-</literal></entry> + <entry align="left"></entry> + </row> <row> <entry align="left"><literal>ALTER VIEW</literal></entry> <entry align="center"><literal>X</literal></entry> @@ -693,6 +701,14 @@ <entry align="center"><literal>-</literal></entry> <entry align="left"></entry> </row> + <row> + <entry align="left"><literal>CREATE VARIABLE</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>-</literal></entry> + <entry align="center"><literal>-</literal></entry> + <entry align="left"></entry> + </row> <row> <entry align="left"><literal>CREATE VIEW</literal></entry> <entry align="center"><literal>X</literal></entry> @@ -981,6 +997,14 @@ <entry align="center"><literal>-</literal></entry> <entry align="left"></entry> </row> + <row> + <entry align="left"><literal>DROP VARIABLE</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>X</literal></entry> + <entry align="center"><literal>-</literal></entry> + <entry align="left"></entry> + </row> <row> <entry align="left"><literal>DROP VIEW</literal></entry> <entry align="center"><literal>X</literal></entry> diff --git a/doc/src/sgml/glossary.sgml b/doc/src/sgml/glossary.sgml index fe8def41d0..44cfa36b96 100644 --- a/doc/src/sgml/glossary.sgml +++ b/doc/src/sgml/glossary.sgml @@ -1598,6 +1598,22 @@ </glossdef> </glossentry> + <glossentry id="glossary-session-variable"> + <glossterm>Session variable</glossterm> + <glossdef> + <para> + A persistent database object that holds a value in session memory. This + memory is not shared across sessions, and after session end, this memory + (the value) is released. The access (read or write) to session variables + is controlled by access rights similarly to other database object access + rights. + </para> + <para> + For more information, see <xref linkend="ddl-session-variables"/>. + </para> + </glossdef> + </glossentry> + <glossentry id="glossary-shared-memory"> <glossterm>Shared memory</glossterm> <glossdef> diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml index f55e901c7e..0f93647382 100644 --- a/doc/src/sgml/plpgsql.sgml +++ b/doc/src/sgml/plpgsql.sgml @@ -5970,6 +5970,18 @@ $$ LANGUAGE plpgsql STRICT IMMUTABLE; </programlisting> </para> </sect3> + + <sect3 id="plpgsql-session-variables"> + <title><command>Session variables</command></title> + + <para> + The <application>PL/pgSQL</application> language has no packages, and + therefore no package variables or package constants. + <productname>PostgreSQL</productname> has session variables and immutable + session variables. Session variables can be created by <command>CREATE + VARIABLE</command>, as described in <xref linkend="sql-createvariable"/>. + </para> + </sect3> </sect2> <sect2 id="plpgsql-porting-appendix"> diff --git a/doc/src/sgml/ref/allfiles.sgml b/doc/src/sgml/ref/allfiles.sgml index 54b5f22d6e..fa295e5d77 100644 --- a/doc/src/sgml/ref/allfiles.sgml +++ b/doc/src/sgml/ref/allfiles.sgml @@ -47,6 +47,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY alterType SYSTEM "alter_type.sgml"> <!ENTITY alterUser SYSTEM "alter_user.sgml"> <!ENTITY alterUserMapping SYSTEM "alter_user_mapping.sgml"> +<!ENTITY alterVariable SYSTEM "alter_variable.sgml"> <!ENTITY alterView SYSTEM "alter_view.sgml"> <!ENTITY analyze SYSTEM "analyze.sgml"> <!ENTITY begin SYSTEM "begin.sgml"> @@ -99,6 +100,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY createType SYSTEM "create_type.sgml"> <!ENTITY createUser SYSTEM "create_user.sgml"> <!ENTITY createUserMapping SYSTEM "create_user_mapping.sgml"> +<!ENTITY createVariable SYSTEM "create_variable.sgml"> <!ENTITY createView SYSTEM "create_view.sgml"> <!ENTITY deallocate SYSTEM "deallocate.sgml"> <!ENTITY declare SYSTEM "declare.sgml"> @@ -147,6 +149,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY dropType SYSTEM "drop_type.sgml"> <!ENTITY dropUser SYSTEM "drop_user.sgml"> <!ENTITY dropUserMapping SYSTEM "drop_user_mapping.sgml"> +<!ENTITY dropVariable SYSTEM "drop_variable.sgml"> <!ENTITY dropView SYSTEM "drop_view.sgml"> <!ENTITY end SYSTEM "end.sgml"> <!ENTITY execute SYSTEM "execute.sgml"> @@ -155,6 +158,7 @@ Complete list of usable sgml source files in this directory. <!ENTITY grant SYSTEM "grant.sgml"> <!ENTITY importForeignSchema SYSTEM "import_foreign_schema.sgml"> <!ENTITY insert SYSTEM "insert.sgml"> +<!ENTITY let SYSTEM "let.sgml"> <!ENTITY listen SYSTEM "listen.sgml"> <!ENTITY load SYSTEM "load.sgml"> <!ENTITY lock SYSTEM "lock.sgml"> diff --git a/doc/src/sgml/ref/alter_default_privileges.sgml b/doc/src/sgml/ref/alter_default_privileges.sgml index a33461fbc2..a4cc0f8907 100644 --- a/doc/src/sgml/ref/alter_default_privileges.sgml +++ b/doc/src/sgml/ref/alter_default_privileges.sgml @@ -50,6 +50,10 @@ GRANT { USAGE | CREATE | ALL [ PRIVILEGES ] } ON SCHEMAS TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] +GRANT { SELECT | UPDATE | ALL [ PRIVILEGES ] } + ON VARIABLES + TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ] + REVOKE [ GRANT OPTION FOR ] { { SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER | MAINTAIN } [, ...] | ALL [ PRIVILEGES ] } @@ -81,6 +85,12 @@ REVOKE [ GRANT OPTION FOR ] ON SCHEMAS FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ CASCADE | RESTRICT ] + +REVOKE [ GRANT OPTION FOR ] + { { SELECT | UPDATE } [, ...] | ALL [ PRIVILEGES ] } + ON VARIABLES + FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] + [ CASCADE | RESTRICT ] </synopsis> </refsynopsisdiv> @@ -92,14 +102,14 @@ REVOKE [ GRANT OPTION FOR ] that will be applied to objects created in the future. (It does not affect privileges assigned to already-existing objects.) Currently, only the privileges for schemas, tables (including views and foreign - tables), sequences, functions, and types (including domains) can be - altered. For this command, functions include aggregates and procedures. - The words <literal>FUNCTIONS</literal> and <literal>ROUTINES</literal> are - equivalent in this command. (<literal>ROUTINES</literal> is preferred - going forward as the standard term for functions and procedures taken - together. In earlier PostgreSQL releases, only the - word <literal>FUNCTIONS</literal> was allowed. It is not possible to set - default privileges for functions and procedures separately.) + tables), sequences, functions, types (including domains) and session + variables can be altered. For this command, functions include aggregates + and procedures. The words <literal>FUNCTIONS</literal> and + <literal>ROUTINES</literal> are equivalent in this command. + (<literal>ROUTINES</literal> is preferred going forward as the standard term + for functions and procedures taken together. In earlier PostgreSQL + releases, only the word <literal>FUNCTIONS</literal> was allowed. It is not + possible to set default privileges for functions and procedures separately.) </para> <para> diff --git a/doc/src/sgml/ref/alter_variable.sgml b/doc/src/sgml/ref/alter_variable.sgml new file mode 100644 index 0000000000..d2036351e5 --- /dev/null +++ b/doc/src/sgml/ref/alter_variable.sgml @@ -0,0 +1,179 @@ +<!-- +doc/src/sgml/ref/alter_variable.sgml +PostgreSQL documentation +--> + +<refentry id="sql-altervariable"> + <indexterm zone="sql-altervariable"> + <primary>ALTER VARIABLE</primary> + </indexterm> + + <indexterm> + <primary>session variable</primary> + <secondary>altering</secondary> + </indexterm> + + <refmeta> + <refentrytitle>ALTER VARIABLE</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo>SQL - Language Statements</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>ALTER VARIABLE</refname> + <refpurpose> + change the definition of a session variable + </refpurpose> + </refnamediv> + + <refsynopsisdiv> +<synopsis> +ALTER VARIABLE <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable class="parameter">new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER } +ALTER VARIABLE <replaceable class="parameter">name</replaceable> RENAME TO <replaceable class="parameter">new_name</replaceable> +ALTER VARIABLE <replaceable class="parameter">name</replaceable> SET SCHEMA <replaceable class="parameter">new_schema</replaceable> +</synopsis> + </refsynopsisdiv> + + <refsect1> + <title>Description</title> + + <para> + The <command>ALTER VARIABLE</command> command changes the definition of an + existing session variable. There are several subforms: + + <variablelist> + <varlistentry> + <term><literal>OWNER</literal></term> + <listitem> + <para> + This form changes the owner of the session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>RENAME</literal></term> + <listitem> + <para> + This form changes the name of the session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>SET SCHEMA</literal></term> + <listitem> + <para> + This form moves the session variable into another schema. + </para> + </listitem> + </varlistentry> + + </variablelist> + </para> + + <para> + Only the owner or a superuser is allowed to alter a session variable. + In order to move a session variable from one schema to another, the user + must also have the <literal>CREATE</literal> privilege on the new schema (or + be a superuser). + + In order to move the session variable ownership from one role to another, + the user must also be a direct or indirect member of the new + owning role, and that role must have the <literal>CREATE</literal> privilege + on the session variable's schema (or be a superuser). These restrictions + enforce that altering the owner doesn't do anything you couldn't do by + dropping and recreating the session variable. + </para> + </refsect1> + + <refsect1> + <title>Parameters</title> + + <para> + <variablelist> + <varlistentry> + <term><replaceable class="parameter">name</replaceable></term> + <listitem> + <para> + The name (possibly schema-qualified) of the existing session variable + to alter. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">new_name</replaceable></term> + <listitem> + <para> + The new name for the session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">new_owner</replaceable></term> + <listitem> + <para> + The user name of the new owner of the session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">new_schema</replaceable></term> + <listitem> + <para> + The new schema for the session variable. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + </refsect1> + + <refsect1> + <title>Examples</title> + + <para> + To rename a session variable: +<programlisting> +ALTER VARIABLE foo RENAME TO boo; +</programlisting> + </para> + + <para> + To change the owner of the session variable <literal>boo</literal> to + <literal>joe</literal>: +<programlisting> +ALTER VARIABLE boo OWNER TO joe; +</programlisting> + </para> + + <para> + To change the schema of the session variable <literal>boo</literal> to + <literal>private</literal>: +<programlisting> +ALTER VARIABLE boo SET SCHEMA private; +</programlisting> + </para> + </refsect1> + + <refsect1> + <title>Compatibility</title> + + <para> + Session variables and this command in particular are a PostgreSQL extension. + </para> + </refsect1> + + <refsect1 id="sql-altervariable-see-also"> + <title>See Also</title> + + <simplelist type="inline"> + <member><xref linkend="sql-createvariable"/></member> + <member><xref linkend="sql-dropvariable"/></member> + <member><xref linkend="sql-let"/></member> + </simplelist> + </refsect1> +</refentry> diff --git a/doc/src/sgml/ref/comment.sgml b/doc/src/sgml/ref/comment.sgml index 5b43c56b13..21cd80818f 100644 --- a/doc/src/sgml/ref/comment.sgml +++ b/doc/src/sgml/ref/comment.sgml @@ -65,6 +65,7 @@ COMMENT ON TRANSFORM FOR <replaceable>type_name</replaceable> LANGUAGE <replaceable>lang_name</replaceable> | TRIGGER <replaceable class="parameter">trigger_name</replaceable> ON <replaceable class="parameter">table_name</replaceable> | TYPE <replaceable class="parameter">object_name</replaceable> | + VARIABLE <replaceable class="parameter">object_name</replaceable> | VIEW <replaceable class="parameter">object_name</replaceable> } IS { <replaceable class="parameter">string_literal</replaceable> | NULL } diff --git a/doc/src/sgml/ref/create_schema.sgml b/doc/src/sgml/ref/create_schema.sgml index ed69298ccc..a834c876bc 100644 --- a/doc/src/sgml/ref/create_schema.sgml +++ b/doc/src/sgml/ref/create_schema.sgml @@ -103,9 +103,10 @@ CREATE SCHEMA IF NOT EXISTS AUTHORIZATION <replaceable class="parameter">role_sp schema. Currently, only <command>CREATE TABLE</command>, <command>CREATE VIEW</command>, <command>CREATE INDEX</command>, <command>CREATE SEQUENCE</command>, <command>CREATE - TRIGGER</command> and <command>GRANT</command> are accepted as clauses - within <command>CREATE SCHEMA</command>. Other kinds of objects may - be created in separate commands after the schema is created. + TRIGGER</command>, <command>GRANT</command> and <command>CREATE + VARIABLE</command> are accepted as clauses within <command>CREATE + SCHEMA</command>. Other kinds of objects may be created in separate + commands after the schema is created. </para> </listitem> </varlistentry> diff --git a/doc/src/sgml/ref/create_variable.sgml b/doc/src/sgml/ref/create_variable.sgml new file mode 100644 index 0000000000..72914e133d --- /dev/null +++ b/doc/src/sgml/ref/create_variable.sgml @@ -0,0 +1,223 @@ +<!-- +doc/src/sgml/ref/create_variable.sgml +PostgreSQL documentation +--> + +<refentry id="sql-createvariable"> + <indexterm zone="sql-createvariable"> + <primary>CREATE VARIABLE</primary> + </indexterm> + + <indexterm> + <primary>session variable</primary> + <secondary>defining</secondary> + </indexterm> + + <refmeta> + <refentrytitle>CREATE VARIABLE</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo>SQL - Language Statements</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>CREATE VARIABLE</refname> + <refpurpose>define a session variable</refpurpose> + </refnamediv> + + <refsynopsisdiv> +<synopsis> +CREATE [ { TEMPORARY | TEMP } ] [ IMMUTABLE ] VARIABLE [ IF NOT EXISTS ] <replaceable class="parameter">name</replaceable> [ AS ] <replaceable class="parameter">data_type</replaceable> ] [ COLLATE <replaceable class="parameter">collation</replaceable> ] + [ NOT NULL ] [ DEFAULT <replaceable class="parameter">default_expr</replaceable> ] [ { ON COMMIT DROP | ON TRANSACTION END RESET } ] +</synopsis> + </refsynopsisdiv> + <refsect1> + <title>Description</title> + + <para> + The <command>CREATE VARIABLE</command> command creates a session variable. + Session variables, like relations, exist within a schema and their access is + controlled via <command>GRANT</command> and <command>REVOKE</command> + commands. + </para> + + <para> + The value of a session variable is local to the current session. Retrieving + a session variable's value returns either a NULL or a default value, unless + its value is set to something else in the current session with a LET + command. The content of a session variable is not transactional. This is the + same as regular variables in PL languages. + </para> + + <para> + Session variables are retrieved by the <command>SELECT</command> SQL + command. Their value is set with the <command>LET</command> SQL command. + While session variables share properties with tables, their value cannot be + changed with an <command>UPDATE</command> command. + </para> + + <note> + <para> + Inside a query or an expression, the session variable can be shadowed by + column or by routine's variable or routine argument. Such collisions of + identifiers can be resolved by using qualified identifiers. Session variables + can use schema name, columns can use table aliases, routine variables + can use block labels, and routine arguments can use the routine name. + </para> + </note> + </refsect1> + + <refsect1> + <title>Parameters</title> + + <variablelist> + <varlistentry id="sql-createvariable-immutable"> + <term><literal>IMMUTABLE</literal></term> + <listitem> + <para> + The assigned value of the session variable can not be changed. + Only if the session variable doesn't have a default value, a single + initialization is allowed using the <command>LET</command> command. Once + done, no further change is allowed until end of transaction + if the session variable was created with clause <literal>ON TRANSACTION + END RESET</literal>, or until reset of all session variables by + <command>DISCARD VARIABLES</command>, or until reset of all session + objects by command <command>DISCARD ALL</command>. + </para> + + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-if-not-exists"> + <term><literal>IF NOT EXISTS</literal></term> + <listitem> + <para> + Do not throw an error if the name already exists. A notice is issued in + this case. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-name"> + <term><replaceable class="parameter">name</replaceable></term> + <listitem> + <para> + The name, optionally schema-qualified, of the session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-data_type"> + <term><replaceable class="parameter">data_type</replaceable></term> + <listitem> + <para> + The name, optionally schema-qualified, of the data type of the session + variable. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-collate"> + <term><literal>COLLATE <replaceable>collation</replaceable></literal></term> + <listitem> + <para> + The <literal>COLLATE</literal> clause assigns a collation to the session + variable (which must be of a collatable data type). If not specified, + the data type's default collation is used. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-not-null"> + <term><literal>NOT NULL</literal></term> + <listitem> + <para> + The <literal>NOT NULL</literal> clause forbids setting the session + variable to a null value. A session variable created as NOT NULL and + without an explicitly declared default value cannot be read until it is + initialized by a LET command. This requires the user to explicitly + initialize the session variable content before reading it, otherwise an + error will be thrown. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-default"> + <term><literal>DEFAULT <replaceable>default_expr</replaceable></literal></term> + <listitem> + <para> + The <literal>DEFAULT</literal> clause can be used to assign a default + value to a session variable. This expression is evaluated when the session + variable is first accessed for reading and had not yet been assigned a value. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-on-commit-drop"> + <term><literal>ON COMMIT DROP</literal></term> + <listitem> + <para> + The <literal>ON COMMIT DROP</literal> clause specifies the behaviour of a + temporary session variable at transaction commit. With this clause, the + session variable is dropped at commit time. The clause is only allowed + for temporary variables. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-createvariable-on-transaction-end-reset"> + <term><literal>ON TRANSACTION END RESET</literal></term> + <listitem> + <para> + The <literal>ON TRANSACTION END RESET</literal> clause causes the session + variable to be reset to its default value when the transaction is committed + or rolled back. + </para> + </listitem> + </varlistentry> + + </variablelist> + </refsect1> + + <refsect1> + <title>Notes</title> + + <para> + Use the <command>DROP VARIABLE</command> command to remove a session + variable. + </para> + </refsect1> + + <refsect1> + <title>Examples</title> + + <para> + Create an date session variable <literal>var1</literal>: +<programlisting> +CREATE VARIABLE var1 AS date; +LET var1 = current_date; +SELECT var1; +</programlisting> + </para> + + </refsect1> + + <refsect1> + <title>Compatibility</title> + + <para> + The <command>CREATE VARIABLE</command> command is a + <productname>PostgreSQL</productname> extension. + </para> + </refsect1> + + <refsect1> + <title>See Also</title> + + <simplelist type="inline"> + <member><xref linkend="sql-altervariable"/></member> + <member><xref linkend="sql-dropvariable"/></member> + <member><xref linkend="sql-let"/></member> + </simplelist> + </refsect1> + +</refentry> diff --git a/doc/src/sgml/ref/discard.sgml b/doc/src/sgml/ref/discard.sgml index bf44c523ca..6f90672afa 100644 --- a/doc/src/sgml/ref/discard.sgml +++ b/doc/src/sgml/ref/discard.sgml @@ -21,7 +21,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> -DISCARD { ALL | PLANS | SEQUENCES | TEMPORARY | TEMP } +DISCARD { ALL | PLANS | SEQUENCES | TEMPORARY | TEMP | VARIABLES } </synopsis> </refsynopsisdiv> @@ -75,6 +75,17 @@ DISCARD { ALL | PLANS | SEQUENCES | TEMPORARY | TEMP } </listitem> </varlistentry> + <varlistentry> + <term><literal>VARIABLES</literal></term> + <listitem> + <para> + Resets the value of all session variables. If a variable + is later reused, it is re-initialized to either + <literal>NULL</literal> or its default value. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>ALL</literal></term> <listitem> @@ -93,6 +104,7 @@ SELECT pg_advisory_unlock_all(); DISCARD PLANS; DISCARD TEMP; DISCARD SEQUENCES; +DISCARD VARIABLES; </programlisting></para> </listitem> </varlistentry> diff --git a/doc/src/sgml/ref/drop_variable.sgml b/doc/src/sgml/ref/drop_variable.sgml new file mode 100644 index 0000000000..67988b5fcd --- /dev/null +++ b/doc/src/sgml/ref/drop_variable.sgml @@ -0,0 +1,118 @@ +<!-- +doc/src/sgml/ref/drop_variable.sgml +PostgreSQL documentation +--> + +<refentry id="sql-dropvariable"> + <indexterm zone="sql-dropvariable"> + <primary>DROP VARIABLE</primary> + </indexterm> + + <indexterm> + <primary>session variable</primary> + <secondary>removing</secondary> + </indexterm> + + <refmeta> + <refentrytitle>DROP VARIABLE</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo>SQL - Language Statements</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>DROP VARIABLE</refname> + <refpurpose>remove a session variable</refpurpose> + </refnamediv> + + <refsynopsisdiv> +<synopsis> +DROP VARIABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable> [, ...] [ CASCADE | RESTRICT ] +</synopsis> + </refsynopsisdiv> + + <refsect1> + <title>Description</title> + + <para> + <command>DROP VARIABLE</command> removes a session variable. + A session variable can only be removed by its owner or a superuser. + </para> + </refsect1> + + <refsect1> + <title>Parameters</title> + + <variablelist> + <varlistentry> + <term><literal>IF EXISTS</literal></term> + <listitem> + <para> + Do not throw an error if the session variable does not exist. A notice is + issued in this case. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><replaceable class="parameter">name</replaceable></term> + <listitem> + <para> + The name, optionally schema-qualified, of a session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>CASCADE</literal></term> + <listitem> + <para> + Automatically drop objects that depend on the session variable (such as + views), and in turn all objects that depend on those objects + (see <xref linkend="ddl-depend"/>). + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>RESTRICT</literal></term> + <listitem> + <para> + Refuse to drop the session variable if any objects depend on it. This is + the default. + </para> + </listitem> + </varlistentry> + </variablelist> + </refsect1> + + <refsect1> + <title>Examples</title> + + <para> + To remove the session variable <literal>var1</literal>: + +<programlisting> +DROP VARIABLE var1; +</programlisting></para> + </refsect1> + + <refsect1> + <title>Compatibility</title> + + <para> + The <command>DROP VARIABLE</command> command is a + <productname>PostgreSQL</productname> extension. + </para> + </refsect1> + + <refsect1> + <title>See Also</title> + + <simplelist type="inline"> + <member><xref linkend="sql-altervariable"/></member> + <member><xref linkend="sql-createvariable"/></member> + <member><xref linkend="sql-let"/></member> + </simplelist> + </refsect1> + +</refentry> diff --git a/doc/src/sgml/ref/grant.sgml b/doc/src/sgml/ref/grant.sgml index 35bf0332c8..ba2a497780 100644 --- a/doc/src/sgml/ref/grant.sgml +++ b/doc/src/sgml/ref/grant.sgml @@ -101,6 +101,12 @@ GRANT <replaceable class="parameter">role_name</replaceable> [, ...] TO <replace [ WITH { ADMIN | INHERIT | SET } { OPTION | TRUE | FALSE } ] [ GRANTED BY <replaceable class="parameter">role_specification</replaceable> ] +GRANT { SELECT | UPDATE | ALL [ PRIVILEGES ] } + ON VARIABLE <replaceable>variable_name</replaceable> [, ...] + | ALL VARIABLES IN SCHEMA <replaceable class="parameter">schema_name</replaceable> [, ...] } + TO <replaceable class="parameter">role_specification</replaceable> [, ...] [ WITH GRANT OPTION ] + [ GRANTED BY <replaceable class="parameter">role_specification</replaceable> ] + <phrase>where <replaceable class="parameter">role_specification</replaceable> can be:</phrase> [ GROUP ] <replaceable class="parameter">role_name</replaceable> diff --git a/doc/src/sgml/ref/let.sgml b/doc/src/sgml/ref/let.sgml new file mode 100644 index 0000000000..c5d1e4d962 --- /dev/null +++ b/doc/src/sgml/ref/let.sgml @@ -0,0 +1,109 @@ +<!-- +doc/src/sgml/ref/let.sgml +PostgreSQL documentation +--> + +<refentry id="sql-let"> + <indexterm zone="sql-let"> + <primary>LET</primary> + </indexterm> + + <indexterm> + <primary>session variable</primary> + <secondary>changing</secondary> + </indexterm> + + <refmeta> + <refentrytitle>LET</refentrytitle> + <manvolnum>7</manvolnum> + <refmiscinfo>SQL - Language Statements</refmiscinfo> + </refmeta> + + <refnamediv> + <refname>LET</refname> + <refpurpose>change a session variable's value</refpurpose> + </refnamediv> + + <refsynopsisdiv> +<synopsis> +LET <replaceable class="parameter">session_variable</replaceable> = <replaceable class="parameter">sql_expression</replaceable> +LET <replaceable class="parameter">session_variable</replaceable> = DEFAULT +</synopsis> + </refsynopsisdiv> + + <refsect1> + <title>Description</title> + + <para> + The <command>LET</command> command assigns a value to the specified session + variable. + </para> + + </refsect1> + + <refsect1> + <title>Parameters</title> + + <variablelist> + <varlistentry> + <term><literal>session_variable</literal></term> + <listitem> + <para> + The name of the session variable. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>sql_expression</literal></term> + <listitem> + <para> + An SQL expression (can be subquery in parenthesis). The result must + be of castable to the same data type as the session variable (in + implicit or assignment context). + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>DEFAULT</literal></term> + <listitem> + <para> + Reset the session variable to its default value, if that is defined. + If no explicit default value has been declared, the session variable + is set to NULL. + </para> + </listitem> + </varlistentry> + </variablelist> + + <para> + Example: +<programlisting> +CREATE VARIABLE myvar AS integer; +LET myvar = 10; +LET myvar = (SELECT sum(val) FROM tab); +LET myvar = DEFAULT; +</programlisting> + </para> + </refsect1> + + <refsect1> + <title>Compatibility</title> + + <para> + The <command>LET</command> is a <productname>PostgreSQL</productname> + extension. + </para> + </refsect1> + + <refsect1> + <title>See Also</title> + + <simplelist type="inline"> + <member><xref linkend="sql-altervariable"/></member> + <member><xref linkend="sql-createvariable"/></member> + <member><xref linkend="sql-dropvariable"/></member> + </simplelist> + </refsect1> +</refentry> diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml index 47bd7dbda0..39cb595647 100644 --- a/doc/src/sgml/ref/pg_restore.sgml +++ b/doc/src/sgml/ref/pg_restore.sgml @@ -106,6 +106,17 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>-A <replaceable class="parameter">session_variable</replaceable></option></term> + <term><option>--variable=<replaceable class="parameter">session_variable</replaceable></option></term> + <listitem> + <para> + Restore a named session variable only. Multiple session variables may + be specified with multiple <option>-A</option> switches. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-c</option></term> <term><option>--clean</option></term> diff --git a/doc/src/sgml/ref/revoke.sgml b/doc/src/sgml/ref/revoke.sgml index 8df492281a..626c0231d0 100644 --- a/doc/src/sgml/ref/revoke.sgml +++ b/doc/src/sgml/ref/revoke.sgml @@ -137,6 +137,13 @@ REVOKE [ { ADMIN | INHERIT | SET } OPTION FOR ] | CURRENT_ROLE | CURRENT_USER | SESSION_USER + +REVOKE [ GRANT OPTION FOR ] + { { SELECT | UPDATE } [, ...] | ALL [ PRIVILEGES ] } + ON VARIABLE <replaceable>variable_name</replaceable> [, ...] + | ALL VARIABLES IN SCHEMA <replaceable class="parameter">schema_name</replaceable> [, ...] } + FROM { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] + [ CASCADE | RESTRICT ] </synopsis> </refsynopsisdiv> diff --git a/doc/src/sgml/reference.sgml b/doc/src/sgml/reference.sgml index e11b4b6130..4256a488f8 100644 --- a/doc/src/sgml/reference.sgml +++ b/doc/src/sgml/reference.sgml @@ -75,6 +75,7 @@ &alterType; &alterUser; &alterUserMapping; + &alterVariable; &alterView; &analyze; &begin; @@ -127,6 +128,7 @@ &createType; &createUser; &createUserMapping; + &createVariable; &createView; &deallocate; &declare; @@ -175,6 +177,7 @@ &dropType; &dropUser; &dropUserMapping; + &dropVariable; &dropView; &end; &execute; @@ -183,6 +186,7 @@ &grant; &importForeignSchema; &insert; + &let; &listen; &load; &lock; -- 2.40.1 [text/x-patch] v20230517-1-0005-DISCARD-VARIABLES-command.patch (3.2K, ../../CAFj8pRAM60yANCnqKHHXZM3j++wpXWOu+rw6tVq5m531LHXiSw@mail.gmail.com/8-v20230517-1-0005-DISCARD-VARIABLES-command.patch) download | inline diff: From bd0c16e0294c69bb234bcd97160fed065f182ce0 Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Tue, 5 Apr 2022 00:13:58 +0200 Subject: [PATCH 05/10] DISCARD VARIABLES command --- src/backend/commands/discard.c | 6 ++++++ src/backend/parser/gram.y | 7 ++++++- src/backend/tcop/utility.c | 3 +++ src/include/nodes/parsenodes.h | 3 ++- src/include/tcop/cmdtaglist.h | 1 + 5 files changed, 18 insertions(+), 2 deletions(-) diff --git a/src/backend/commands/discard.c b/src/backend/commands/discard.c index 296dc82d2e..a8189dd12c 100644 --- a/src/backend/commands/discard.c +++ b/src/backend/commands/discard.c @@ -19,6 +19,7 @@ #include "commands/discard.h" #include "commands/prepare.h" #include "commands/sequence.h" +#include "commands/session_variable.h" #include "utils/guc.h" #include "utils/portal.h" @@ -48,6 +49,10 @@ DiscardCommand(DiscardStmt *stmt, bool isTopLevel) ResetTempTableNamespace(); break; + case DISCARD_VARIABLES: + ResetSessionVariables(); + break; + default: elog(ERROR, "unrecognized DISCARD target: %d", stmt->target); } @@ -75,4 +80,5 @@ DiscardAll(bool isTopLevel) ResetPlanCache(); ResetTempTableNamespace(); ResetSequenceCaches(); + ResetSessionVariables(); } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 3c6faa42a6..60401ef05d 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -2071,7 +2071,12 @@ DiscardStmt: n->target = DISCARD_SEQUENCES; $$ = (Node *) n; } - + | DISCARD VARIABLES + { + DiscardStmt *n = makeNode(DiscardStmt); + n->target = DISCARD_VARIABLES; + $$ = (Node *) n; + } ; diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 40cc8231fb..df322e4fcb 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -2962,6 +2962,9 @@ CreateCommandTag(Node *parsetree) case DISCARD_SEQUENCES: tag = CMDTAG_DISCARD_SEQUENCES; break; + case DISCARD_VARIABLES: + tag = CMDTAG_DISCARD_VARIABLES; + break; default: tag = CMDTAG_UNKNOWN; } diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index b96b56fd6a..8948523140 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -3794,7 +3794,8 @@ typedef enum DiscardMode DISCARD_ALL, DISCARD_PLANS, DISCARD_SEQUENCES, - DISCARD_TEMP + DISCARD_TEMP, + DISCARD_VARIABLES } DiscardMode; typedef struct DiscardStmt diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h index 648a4af305..8ce00b4ea1 100644 --- a/src/include/tcop/cmdtaglist.h +++ b/src/include/tcop/cmdtaglist.h @@ -135,6 +135,7 @@ PG_CMDTAG(CMDTAG_DISCARD_ALL, "DISCARD ALL", false, false, false) PG_CMDTAG(CMDTAG_DISCARD_PLANS, "DISCARD PLANS", false, false, false) PG_CMDTAG(CMDTAG_DISCARD_SEQUENCES, "DISCARD SEQUENCES", false, false, false) PG_CMDTAG(CMDTAG_DISCARD_TEMP, "DISCARD TEMP", false, false, false) +PG_CMDTAG(CMDTAG_DISCARD_VARIABLES, "DISCARD VARIABLES", false, false, false) PG_CMDTAG(CMDTAG_DO, "DO", false, false, false) PG_CMDTAG(CMDTAG_DROP_ACCESS_METHOD, "DROP ACCESS METHOD", true, false, false) PG_CMDTAG(CMDTAG_DROP_AGGREGATE, "DROP AGGREGATE", true, false, false) -- 2.40.1 [text/x-patch] v20230517-1-0004-support-of-LET-command-in-PLpgSQL.patch (11.9K, ../../CAFj8pRAM60yANCnqKHHXZM3j++wpXWOu+rw6tVq5m531LHXiSw@mail.gmail.com/9-v20230517-1-0004-support-of-LET-command-in-PLpgSQL.patch) download | inline diff: From 5d7684425ba993371a4fe1608ffdc39be27be666 Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Mon, 4 Apr 2022 20:32:45 +0200 Subject: [PATCH 04/10] support of LET command in PLpgSQL The support for LET command as PLpgSQL command does assigning of expression evaluation to session variable. With extra support in PLpgSQL we can implement SQL LET command like usual PostgreSQL's command (and the assigning of result to session variable is not side effect of query evaluation). --- src/backend/executor/spi.c | 3 ++ src/backend/parser/gram.y | 8 ++++ src/backend/parser/parser.c | 3 +- src/include/parser/parser.h | 6 ++- src/pl/plpgsql/src/pl_exec.c | 55 +++++++++++++++++++++++++ src/pl/plpgsql/src/pl_funcs.c | 24 +++++++++++ src/pl/plpgsql/src/pl_gram.y | 28 ++++++++++++- src/pl/plpgsql/src/pl_reserved_kwlist.h | 1 + src/pl/plpgsql/src/plpgsql.h | 14 ++++++- src/tools/pgindent/typedefs.list | 1 + 10 files changed, 139 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/spi.c b/src/backend/executor/spi.c index 256632c985..f11df000e7 100644 --- a/src/backend/executor/spi.c +++ b/src/backend/executor/spi.c @@ -2967,6 +2967,9 @@ _SPI_error_callback(void *arg) case RAW_PARSE_PLPGSQL_ASSIGN3: errcontext("PL/pgSQL assignment \"%s\"", query); break; + case RAW_PARSE_PLPGSQL_LET: + errcontext("LET statement \"%s\"", query); + break; default: errcontext("SQL statement \"%s\"", query); break; diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 8ad66520bb..3c6faa42a6 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -814,6 +814,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %token MODE_PLPGSQL_ASSIGN1 %token MODE_PLPGSQL_ASSIGN2 %token MODE_PLPGSQL_ASSIGN3 +%token MODE_PLPGSQL_LET /* Precedence: lowest to highest */ @@ -929,6 +930,13 @@ parse_toplevel: pg_yyget_extra(yyscanner)->parsetree = list_make1(makeRawStmt((Node *) n, 0)); } + | MODE_PLPGSQL_LET LetStmt + { + LetStmt *n = (LetStmt *) $2; + n->plpgsql_mode = true; + pg_yyget_extra(yyscanner)->parsetree = + list_make1(makeRawStmt((Node *) n, 0)); + } ; /* diff --git a/src/backend/parser/parser.c b/src/backend/parser/parser.c index e17c310cc1..d58f06342a 100644 --- a/src/backend/parser/parser.c +++ b/src/backend/parser/parser.c @@ -61,7 +61,8 @@ raw_parser(const char *str, RawParseMode mode) MODE_PLPGSQL_EXPR, /* RAW_PARSE_PLPGSQL_EXPR */ MODE_PLPGSQL_ASSIGN1, /* RAW_PARSE_PLPGSQL_ASSIGN1 */ MODE_PLPGSQL_ASSIGN2, /* RAW_PARSE_PLPGSQL_ASSIGN2 */ - MODE_PLPGSQL_ASSIGN3 /* RAW_PARSE_PLPGSQL_ASSIGN3 */ + MODE_PLPGSQL_ASSIGN3, /* RAW_PARSE_PLPGSQL_ASSIGN3 */ + MODE_PLPGSQL_LET /* RAW_PARSE_PLPGSQL_LET */ }; yyextra.have_lookahead = true; diff --git a/src/include/parser/parser.h b/src/include/parser/parser.h index 8d90064d87..0548ad8370 100644 --- a/src/include/parser/parser.h +++ b/src/include/parser/parser.h @@ -33,6 +33,9 @@ * RAW_PARSE_PLPGSQL_ASSIGNn: parse a PL/pgSQL assignment statement, * and return a one-element List containing a RawStmt node. "n" * gives the number of dotted names comprising the target ColumnRef. + * + * RAW_PARSE_PLPGSQL_LET: parse a LET statement, and return a + * one-element List containing a RawStmt node. */ typedef enum { @@ -41,7 +44,8 @@ typedef enum RAW_PARSE_PLPGSQL_EXPR, RAW_PARSE_PLPGSQL_ASSIGN1, RAW_PARSE_PLPGSQL_ASSIGN2, - RAW_PARSE_PLPGSQL_ASSIGN3 + RAW_PARSE_PLPGSQL_ASSIGN3, + RAW_PARSE_PLPGSQL_LET } RawParseMode; /* Values for the backslash_quote GUC */ diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c index 4b76f7699a..583f84e8ea 100644 --- a/src/pl/plpgsql/src/pl_exec.c +++ b/src/pl/plpgsql/src/pl_exec.c @@ -24,6 +24,7 @@ #include "catalog/pg_proc.h" #include "catalog/pg_type.h" #include "commands/defrem.h" +#include "commands/session_variable.h" #include "executor/execExpr.h" #include "executor/spi.h" #include "executor/tstoreReceiver.h" @@ -328,6 +329,8 @@ static int exec_stmt_commit(PLpgSQL_execstate *estate, PLpgSQL_stmt_commit *stmt); static int exec_stmt_rollback(PLpgSQL_execstate *estate, PLpgSQL_stmt_rollback *stmt); +static int exec_stmt_let(PLpgSQL_execstate *estate, + PLpgSQL_stmt_let *let); static void plpgsql_estate_setup(PLpgSQL_execstate *estate, PLpgSQL_function *func, @@ -2119,6 +2122,10 @@ exec_stmts(PLpgSQL_execstate *estate, List *stmts) rc = exec_stmt_rollback(estate, (PLpgSQL_stmt_rollback *) stmt); break; + case PLPGSQL_STMT_LET: + rc = exec_stmt_let(estate, (PLpgSQL_stmt_let *) stmt); + break; + default: /* point err_stmt to parent, since this one seems corrupt */ estate->err_stmt = save_estmt; @@ -4998,6 +5005,54 @@ exec_stmt_rollback(PLpgSQL_execstate *estate, PLpgSQL_stmt_rollback *stmt) return PLPGSQL_RC_OK; } +/* ---------- + * exec_stmt_let Evaluate an expression and + * put the result into a session variable. + * ---------- + */ +static int +exec_stmt_let(PLpgSQL_execstate *estate, PLpgSQL_stmt_let *stmt) +{ + bool isNull; + Oid valtype; + int32 valtypmod; + Datum value; + Oid varid; + + List *plansources; + CachedPlanSource *plansource; + + value = exec_eval_expr(estate, + stmt->expr, + &isNull, + &valtype, + &valtypmod); + + /* + * Oid of target session variable is stored in Query structure. It is + * safer to read this value directly from the plan than to hold this value + * in the plpgsql context, because it's not necessary to handle + * invalidation of the cached value. Next operations are read only without + * any allocations, so we can expect that retrieving varid from Query + * should be fast. + */ + plansources = SPI_plan_get_plan_sources(stmt->expr->plan); + if (list_length(plansources) != 1) + elog(ERROR, "unexpected length of plansources of query for LET statement"); + + plansource = (CachedPlanSource *) linitial(plansources); + if (list_length(plansource->query_list) != 1) + elog(ERROR, "unexpected length of plansource of query for LET statement"); + + varid = linitial_node(Query, plansource->query_list)->resultVariable; + if (!OidIsValid(varid)) + elog(ERROR, "oid of target session variable is not valid"); + + SetSessionVariableWithSecurityCheck(varid, value, isNull); + + return PLPGSQL_RC_OK; +} + /* ---------- * exec_assign_expr Put an expression's result into a variable. * ---------- diff --git a/src/pl/plpgsql/src/pl_funcs.c b/src/pl/plpgsql/src/pl_funcs.c index f010515fdf..73142a9147 100644 --- a/src/pl/plpgsql/src/pl_funcs.c +++ b/src/pl/plpgsql/src/pl_funcs.c @@ -288,6 +288,8 @@ plpgsql_stmt_typename(PLpgSQL_stmt *stmt) return "COMMIT"; case PLPGSQL_STMT_ROLLBACK: return "ROLLBACK"; + case PLPGSQL_STMT_LET: + return "LET"; } return "unknown"; @@ -370,6 +372,7 @@ static void free_perform(PLpgSQL_stmt_perform *stmt); static void free_call(PLpgSQL_stmt_call *stmt); static void free_commit(PLpgSQL_stmt_commit *stmt); static void free_rollback(PLpgSQL_stmt_rollback *stmt); +static void free_let(PLpgSQL_stmt_let *stmt); static void free_expr(PLpgSQL_expr *expr); @@ -459,6 +462,9 @@ free_stmt(PLpgSQL_stmt *stmt) case PLPGSQL_STMT_ROLLBACK: free_rollback((PLpgSQL_stmt_rollback *) stmt); break; + case PLPGSQL_STMT_LET: + free_let((PLpgSQL_stmt_let *) stmt); + break; default: elog(ERROR, "unrecognized cmd_type: %d", stmt->cmd_type); break; @@ -713,6 +719,12 @@ free_getdiag(PLpgSQL_stmt_getdiag *stmt) { } +static void +free_let(PLpgSQL_stmt_let *stmt) +{ + free_expr(stmt->expr); +} + static void free_expr(PLpgSQL_expr *expr) { @@ -815,6 +827,7 @@ static void dump_perform(PLpgSQL_stmt_perform *stmt); static void dump_call(PLpgSQL_stmt_call *stmt); static void dump_commit(PLpgSQL_stmt_commit *stmt); static void dump_rollback(PLpgSQL_stmt_rollback *stmt); +static void dump_let(PLpgSQL_stmt_let *stmt); static void dump_expr(PLpgSQL_expr *expr); @@ -914,6 +927,9 @@ dump_stmt(PLpgSQL_stmt *stmt) case PLPGSQL_STMT_ROLLBACK: dump_rollback((PLpgSQL_stmt_rollback *) stmt); break; + case PLPGSQL_STMT_LET: + dump_let((PLpgSQL_stmt_let *) stmt); + break; default: elog(ERROR, "unrecognized cmd_type: %d", stmt->cmd_type); break; @@ -1590,6 +1606,14 @@ dump_getdiag(PLpgSQL_stmt_getdiag *stmt) printf("\n"); } +static void +dump_let(PLpgSQL_stmt_let *stmt) +{ + dump_ind(); + dump_expr(stmt->expr); + printf("\n"); +} + static void dump_expr(PLpgSQL_expr *expr) { diff --git a/src/pl/plpgsql/src/pl_gram.y b/src/pl/plpgsql/src/pl_gram.y index 6a09bfdd67..8aa6b5d7ab 100644 --- a/src/pl/plpgsql/src/pl_gram.y +++ b/src/pl/plpgsql/src/pl_gram.y @@ -195,7 +195,7 @@ static void check_raise_parameters(PLpgSQL_stmt_raise *stmt); %type <stmt> stmt_return stmt_raise stmt_assert stmt_execsql %type <stmt> stmt_dynexecute stmt_for stmt_perform stmt_call stmt_getdiag %type <stmt> stmt_open stmt_fetch stmt_move stmt_close stmt_null -%type <stmt> stmt_commit stmt_rollback +%type <stmt> stmt_commit stmt_rollback stmt_let %type <stmt> stmt_case stmt_foreach_a %type <list> proc_exceptions @@ -302,6 +302,7 @@ static void check_raise_parameters(PLpgSQL_stmt_raise *stmt); %token <keyword> K_INTO %token <keyword> K_IS %token <keyword> K_LAST +%token <keyword> K_LET %token <keyword> K_LOG %token <keyword> K_LOOP %token <keyword> K_MERGE @@ -870,6 +871,8 @@ proc_stmt : pl_block ';' { $$ = $1; } | stmt_rollback { $$ = $1; } + | stmt_let + { $$ = $1; } ; stmt_perform : K_PERFORM @@ -986,6 +989,29 @@ stmt_assign : T_DATUM } ; +stmt_let : K_LET + { + PLpgSQL_stmt_let *new; + RawParseMode pmode; + + pmode = RAW_PARSE_PLPGSQL_LET; + + new = palloc0(sizeof(PLpgSQL_stmt_let)); + new->cmd_type = PLPGSQL_STMT_LET; + new->lineno = plpgsql_location_to_lineno(@1); + new->stmtid = ++plpgsql_curr_compile->nstatements; + + /* Push back the head name to include it in the stmt */ + plpgsql_push_back_token(K_LET); + new->expr = read_sql_construct(';', 0, 0, ";", + pmode, + false, true, true, + NULL, NULL); + + $$ = (PLpgSQL_stmt *)new; + } + ; + stmt_getdiag : K_GET getdiag_area_opt K_DIAGNOSTICS getdiag_list ';' { PLpgSQL_stmt_getdiag *new; diff --git a/src/pl/plpgsql/src/pl_reserved_kwlist.h b/src/pl/plpgsql/src/pl_reserved_kwlist.h index c116abbb7a..90eb1ca8aa 100644 --- a/src/pl/plpgsql/src/pl_reserved_kwlist.h +++ b/src/pl/plpgsql/src/pl_reserved_kwlist.h @@ -40,6 +40,7 @@ PG_KEYWORD("from", K_FROM) PG_KEYWORD("if", K_IF) PG_KEYWORD("in", K_IN) PG_KEYWORD("into", K_INTO) +PG_KEYWORD("let", K_LET) PG_KEYWORD("loop", K_LOOP) PG_KEYWORD("not", K_NOT) PG_KEYWORD("null", K_NULL) diff --git a/src/pl/plpgsql/src/plpgsql.h b/src/pl/plpgsql/src/plpgsql.h index 2b4bcd1dbe..22c9d78f30 100644 --- a/src/pl/plpgsql/src/plpgsql.h +++ b/src/pl/plpgsql/src/plpgsql.h @@ -127,7 +127,8 @@ typedef enum PLpgSQL_stmt_type PLPGSQL_STMT_PERFORM, PLPGSQL_STMT_CALL, PLPGSQL_STMT_COMMIT, - PLPGSQL_STMT_ROLLBACK + PLPGSQL_STMT_ROLLBACK, + PLPGSQL_STMT_LET } PLpgSQL_stmt_type; /* @@ -520,6 +521,17 @@ typedef struct PLpgSQL_stmt_assign PLpgSQL_expr *expr; } PLpgSQL_stmt_assign; +/* + * Let statement + */ +typedef struct PLpgSQL_stmt_let +{ + PLpgSQL_stmt_type cmd_type; + int lineno; + unsigned int stmtid; + PLpgSQL_expr *expr; +} PLpgSQL_stmt_let; + /* * PERFORM statement */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 5053ab081e..429fb8cdcf 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1796,6 +1796,7 @@ PLpgSQL_stmt_forq PLpgSQL_stmt_fors PLpgSQL_stmt_getdiag PLpgSQL_stmt_if +PLpgSQL_stmt_let PLpgSQL_stmt_loop PLpgSQL_stmt_open PLpgSQL_stmt_perform -- 2.40.1 [text/x-patch] v20230517-1-0003-LET-command.patch (43.7K, ../../CAFj8pRAM60yANCnqKHHXZM3j++wpXWOu+rw6tVq5m531LHXiSw@mail.gmail.com/10-v20230517-1-0003-LET-command.patch) download | inline diff: From ba280f0a9319d6c476419073f34210a0f85bbbac Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Thu, 10 Nov 2022 22:03:04 +0100 Subject: [PATCH 03/10] LET command Set session variable value to result of query or result of default expression --- src/backend/commands/explain.c | 16 ++ src/backend/commands/session_variable.c | 86 +++++++ src/backend/executor/Makefile | 1 + src/backend/executor/execMain.c | 1 + src/backend/executor/meson.build | 1 + src/backend/executor/svariableReceiver.c | 214 +++++++++++++++++ src/backend/nodes/nodeFuncs.c | 10 + src/backend/optimizer/plan/setrefs.c | 38 ++- src/backend/parser/analyze.c | 292 +++++++++++++++++++++++ src/backend/parser/gram.y | 57 ++++- src/backend/parser/parse_agg.c | 4 + src/backend/parser/parse_cte.c | 8 + src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + src/backend/parser/parse_target.c | 4 +- src/backend/rewrite/rewriteHandler.c | 34 +++ src/backend/tcop/dest.c | 7 + src/backend/tcop/utility.c | 15 ++ src/backend/utils/cache/plancache.c | 12 + src/include/commands/session_variable.h | 3 + src/include/executor/svariableReceiver.h | 25 ++ src/include/nodes/parsenodes.h | 18 ++ src/include/nodes/plannodes.h | 2 +- src/include/parser/kwlist.h | 1 + src/include/parser/parse_node.h | 1 + src/include/tcop/cmdtaglist.h | 1 + src/include/tcop/dest.h | 3 +- src/tools/pgindent/typedefs.list | 2 + 28 files changed, 843 insertions(+), 20 deletions(-) create mode 100644 src/backend/executor/svariableReceiver.c create mode 100644 src/include/executor/svariableReceiver.h diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index 5334c503e1..630116a2e2 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -501,6 +501,22 @@ ExplainOneUtility(Node *utilityStmt, IntoClause *into, ExplainState *es, else ExplainDummyGroup("Notify", NULL, es); } + else if (IsA(utilityStmt, LetStmt)) + { + LetStmt *letstmt = (LetStmt *) utilityStmt; + List *rewritten; + + if (es->format == EXPLAIN_FORMAT_TEXT) + appendStringInfoString(es->str, "SET SESSION VARIABLE\n"); + else + ExplainDummyGroup("Set Session Variable", NULL, es); + + rewritten = QueryRewrite(castNode(Query, copyObject(letstmt->query))); + Assert(list_length(rewritten) == 1); + ExplainOneQuery(linitial_node(Query, rewritten), + CURSOR_OPT_PARALLEL_OK, NULL, es, + queryString, params, queryEnv); + } else { if (es->format == EXPLAIN_FORMAT_TEXT) diff --git a/src/backend/commands/session_variable.c b/src/backend/commands/session_variable.c index b4a91ea5a3..7e8455b26a 100644 --- a/src/backend/commands/session_variable.c +++ b/src/backend/commands/session_variable.c @@ -20,16 +20,20 @@ #include "catalog/namespace.h" #include "catalog/pg_variable.h" #include "commands/session_variable.h" +#include "executor/svariableReceiver.h" #include "funcapi.h" #include "miscadmin.h" #include "optimizer/optimizer.h" +#include "rewrite/rewriteHandler.h" #include "storage/lmgr.h" #include "storage/proc.h" +#include "tcop/tcopprot.h" #include "utils/builtins.h" #include "utils/datum.h" #include "utils/inval.h" #include "utils/lsyscache.h" #include "utils/memutils.h" +#include "utils/snapmgr.h" #include "utils/syscache.h" /* @@ -1240,6 +1244,88 @@ AtEOSubXact_SessionVariable(bool isCommit, } } +/* + * Assign result of evaluated expression to session variable + */ +void +ExecuteLetStmt(ParseState *pstate, + LetStmt *stmt, + ParamListInfo params, + QueryEnvironment *queryEnv, + QueryCompletion *qc) +{ + Query *query = castNode(Query, stmt->query); + List *rewritten; + DestReceiver *dest; + AclResult aclresult; + PlannedStmt *plan; + QueryDesc *queryDesc; + Oid varid = query->resultVariable; + + Assert(OidIsValid(varid)); + + /* + * Is it allowed to write to session variable? + */ + aclresult = object_aclcheck(VariableRelationId, varid, GetUserId(), ACL_UPDATE); + if (aclresult != ACLCHECK_OK) + aclcheck_error(aclresult, OBJECT_VARIABLE, get_session_variable_name(varid)); + + /* Create dest receiver for LET */ + dest = CreateDestReceiver(DestVariable); + SetVariableDestReceiverVarid(dest, varid); + + /* run rewriter - can be used for replacement of DEFAULT node */ + query = copyObject(query); + + rewritten = QueryRewrite(query); + + Assert(list_length(rewritten) == 1); + + query = linitial_node(Query, rewritten); + Assert(query->commandType == CMD_SELECT); + + /* plan the query */ + plan = pg_plan_query(query, pstate->p_sourcetext, + CURSOR_OPT_PARALLEL_OK, params); + + /* + * Use a snapshot with an updated command ID to ensure this query sees + * results of any previously executed queries. (This could only matter if + * the planner executed an allegedly-stable function that changed the + * database contents, but let's do it anyway to be parallel to the EXPLAIN + * code path.) + */ + PushCopiedSnapshot(GetActiveSnapshot()); + UpdateActiveSnapshotCommandId(); + + /* Create a QueryDesc, redirecting output to our tuple receiver */ + queryDesc = CreateQueryDesc(plan, pstate->p_sourcetext, + GetActiveSnapshot(), InvalidSnapshot, + dest, params, queryEnv, 0); + + /* call ExecutorStart to prepare the plan for execution */ + ExecutorStart(queryDesc, 0); + + /* + * run the plan to completion. The result should be only one + * row. For an check too_many_rows we need to read two rows. + */ + ExecutorRun(queryDesc, ForwardScanDirection, 2L, true); + + /* save the rowcount if we're given a qc to fill */ + if (qc) + SetQueryCompletion(qc, CMDTAG_LET, queryDesc->estate->es_processed); + + /* and clean up */ + ExecutorFinish(queryDesc); + ExecutorEnd(queryDesc); + + FreeQueryDesc(queryDesc); + + PopActiveSnapshot(); +} + /* * pg_session_variables - designed for testing * diff --git a/src/backend/executor/Makefile b/src/backend/executor/Makefile index 11118d0ce0..71248a34f2 100644 --- a/src/backend/executor/Makefile +++ b/src/backend/executor/Makefile @@ -76,6 +76,7 @@ OBJS = \ nodeWindowAgg.o \ nodeWorktablescan.o \ spi.o \ + svariableReceiver.o \ tqueue.o \ tstoreReceiver.o diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index a407b53038..0be24d2554 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -51,6 +51,7 @@ #include "commands/session_variable.h" #include "executor/execdebug.h" #include "executor/nodeSubplan.h" +#include "executor/svariableReceiver.h" #include "foreign/fdwapi.h" #include "jit/jit.h" #include "mb/pg_wchar.h" diff --git a/src/backend/executor/meson.build b/src/backend/executor/meson.build index 65f9457c9b..b34b383b07 100644 --- a/src/backend/executor/meson.build +++ b/src/backend/executor/meson.build @@ -64,6 +64,7 @@ backend_sources += files( 'nodeWindowAgg.c', 'nodeWorktablescan.c', 'spi.c', + 'svariableReceiver.c', 'tqueue.c', 'tstoreReceiver.c', ) diff --git a/src/backend/executor/svariableReceiver.c b/src/backend/executor/svariableReceiver.c new file mode 100644 index 0000000000..7c975cbbf9 --- /dev/null +++ b/src/backend/executor/svariableReceiver.c @@ -0,0 +1,214 @@ +/*------------------------------------------------------------------------- + * + * svariableReceiver.c + * An implementation of DestReceiver that stores the result value in + * a session variable. + * + * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/backend/executor/svariableReceiver.c + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" +#include "miscadmin.h" + +#include "access/detoast.h" +#include "executor/svariableReceiver.h" +#include "commands/session_variable.h" +#include "storage/lock.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" + +typedef struct +{ + DestReceiver pub; + Oid varid; + Oid typid; + int32 typmod; + int typlen; + int slot_offset; + int rows; +} SVariableState; + + +/* + * Prepare to receive tuples from executor. + */ +static void +svariableStartupReceiver(DestReceiver *self, int operation, TupleDesc typeinfo) +{ + SVariableState *myState = (SVariableState *) self; + int natts = typeinfo->natts; + int outcols = 0; + int i; + + /* Receiver should be initialized by SetVariableDestReceiverVarid */ + Assert(OidIsValid(myState->varid)); + + for (i = 0; i < natts; i++) + { + Form_pg_attribute attr = TupleDescAttr(typeinfo, i); + Oid typid; + Oid collid; + int32 typmod; + + if (attr->attisdropped) + continue; + + if (++outcols > 1) + continue; + + get_session_variable_type_typmod_collid(myState->varid, + &typid, + &typmod, + &collid); + + /* + * double check - the type and typmod of target variable should be + * same as type and typmod of assignment expression. It should be, the + * expression is wrapped by cast to target type and typmod. + */ + if (attr->atttypid != typid || + (attr->atttypmod >= 0 && + attr->atttypmod != typmod)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("target session variable is of type %s" + " but expression is of type %s", + format_type_with_typemod(typid, typmod), + format_type_with_typemod(attr->atttypid, + attr->atttypmod)))); + + myState->typid = attr->atttypid; + myState->typmod = attr->atttypmod; + myState->typlen = attr->attlen; + myState->slot_offset = i; + } + + if (outcols != 1) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg_plural("assignment expression returned %d column", + "assignment expression returned %d columns", + outcols, + outcols))); + + + + myState->rows = 0; +} + +/* + * Receive a tuple from the executor and store it in session variable. + */ +static bool +svariableReceiveSlot(TupleTableSlot *slot, DestReceiver *self) +{ + SVariableState *myState = (SVariableState *) self; + Datum value; + bool isnull; + bool freeval = false; + + /* Make sure the tuple is fully deconstructed */ + slot_getallattrs(slot); + + value = slot->tts_values[myState->slot_offset]; + isnull = slot->tts_isnull[myState->slot_offset]; + + if (myState->typlen == -1 && !isnull && VARATT_IS_EXTERNAL(DatumGetPointer(value))) + { + value = PointerGetDatum(detoast_external_attr((struct varlena *) + DatumGetPointer(value))); + freeval = true; + } + + myState->rows += 1; + + if (myState->rows > 1) + ereport(ERROR, + (errcode(ERRCODE_TOO_MANY_ROWS), + errmsg("expression returned more than one row"))); + + SetSessionVariable(myState->varid, value, isnull); + + if (freeval) + pfree(DatumGetPointer(value)); + + return true; +} + +/* + * Clean up at end of an executor run + */ +static void +svariableShutdownReceiver(DestReceiver *self) +{ + if (((SVariableState *) self)->rows == 0) + ereport(ERROR, + (errcode(ERRCODE_NO_DATA_FOUND), + errmsg("expression returned no rows"))); +} + +/* + * Destroy receiver when done with it + */ +static void +svariableDestroyReceiver(DestReceiver *self) +{ + pfree(self); +} + +/* + * Initially create a DestReceiver object. + */ +DestReceiver * +CreateVariableDestReceiver(void) +{ + SVariableState *self = (SVariableState *) palloc0(sizeof(SVariableState)); + + self->pub.receiveSlot = svariableReceiveSlot; + self->pub.rStartup = svariableStartupReceiver; + self->pub.rShutdown = svariableShutdownReceiver; + self->pub.rDestroy = svariableDestroyReceiver; + self->pub.mydest = DestVariable; + + /* + * Private fields will be set by SetVariableDestReceiverVarid and + * svariableStartupReceiver. + */ + return (DestReceiver *) self; +} + +/* + * Set parameters for a VariableDestReceiver. + * Should be called right after creating the DestReceiver. + */ +void +SetVariableDestReceiverVarid(DestReceiver *self, Oid varid) +{ + SVariableState *myState = (SVariableState *) self; + LOCKTAG locktag PG_USED_FOR_ASSERTS_ONLY; + + Assert(myState->pub.mydest == DestVariable); + Assert(OidIsValid(varid)); + Assert(SearchSysCacheExists1(VARIABLEOID, varid)); + +#ifdef USE_ASSERT_CHECKING + + SET_LOCKTAG_OBJECT(locktag, + MyDatabaseId, + VariableRelationId, + varid, + 0); + + Assert(LockHeldByMe(&locktag, AccessShareLock)); + +#endif + + myState->varid = varid; +} diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 0ed8712a63..25c3d5d825 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -4073,6 +4073,16 @@ raw_expression_tree_walker_impl(Node *node, return true; } break; + case T_LetStmt: + { + LetStmt *stmt = (LetStmt *) node; + + if (WALK(stmt->target)) + return true; + if (WALK(stmt->query)) + return true; + } + break; case T_PLAssignStmt: { PLAssignStmt *stmt = (PLAssignStmt *) node; diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index f18ed8ef9a..fadec889e2 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -213,7 +213,7 @@ static List *set_windowagg_runcondition_references(PlannerInfo *root, Plan *plan); static bool pull_up_has_session_variables_walker(Node *node, PlannerInfo *root); - +static void record_plan_variable_dependency(PlannerInfo *root, Oid varid); /***************************************************************************** * @@ -2074,18 +2074,7 @@ fix_expr_common(PlannerInfo *root, Node *node) Param *p = (Param *) node; if (p->paramkind == PARAM_VARIABLE) - { - PlanInvalItem *inval_item = makeNode(PlanInvalItem); - - /* paramid is still session variable id */ - inval_item->cacheId = VARIABLEOID; - inval_item->hashValue = GetSysCacheHashValue1(VARIABLEOID, - ObjectIdGetDatum(p->paramvarid)); - - /* Append this variable to global, register dependency */ - root->glob->invalItems = lappend(root->glob->invalItems, - inval_item); - } + record_plan_variable_dependency(root, p->paramvarid); } } @@ -3576,6 +3565,25 @@ record_plan_type_dependency(PlannerInfo *root, Oid typid) } } +/* + * Record dependency on a session variable. The variable can be used as a + * session variable in expression list, or target of LET statement. + */ +static void +record_plan_variable_dependency(PlannerInfo *root, Oid varid) +{ + PlanInvalItem *inval_item = makeNode(PlanInvalItem); + + /* paramid is still session variable id */ + inval_item->cacheId = VARIABLEOID; + inval_item->hashValue = GetSysCacheHashValue1(VARIABLEOID, + ObjectIdGetDatum(varid)); + + /* Append this variable to global, register dependency */ + root->glob->invalItems = lappend(root->glob->invalItems, + inval_item); +} + /* * extract_query_dependencies * Given a rewritten, but not yet planned, query or queries @@ -3649,6 +3657,10 @@ extract_query_dependencies_walker(Node *node, PlannerInfo *context) query = UtilityContainsQuery(query->utilityStmt); if (query == NULL) return false; + + /* Record the session variable used as target of LET statement. */ + if (OidIsValid(query->resultVariable)) + record_plan_variable_dependency(context, query->resultVariable); } /* Remember if any Query has RLS quals applied by rewriter */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index c2f4f4836f..fd17b4f7c0 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -25,9 +25,12 @@ #include "postgres.h" #include "access/sysattr.h" +#include "catalog/namespace.h" #include "catalog/pg_proc.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "commands/defrem.h" +#include "commands/session_variable.h" #include "miscadmin.h" #include "nodes/makefuncs.h" #include "nodes/nodeFuncs.h" @@ -52,6 +55,7 @@ #include "utils/backend_status.h" #include "utils/builtins.h" #include "utils/guc.h" +#include "utils/lsyscache.h" #include "utils/rel.h" #include "utils/syscache.h" @@ -85,6 +89,8 @@ static Query *transformCreateTableAsStmt(ParseState *pstate, CreateTableAsStmt *stmt); static Query *transformCallStmt(ParseState *pstate, CallStmt *stmt); +static Query *transformLetStmt(ParseState *pstate, + LetStmt *stmt); static void transformLockingClause(ParseState *pstate, Query *qry, LockingClause *lc, bool pushedDown); #ifdef RAW_EXPRESSION_COVERAGE_TEST @@ -328,6 +334,7 @@ transformStmt(ParseState *pstate, Node *parseTree) case T_UpdateStmt: case T_DeleteStmt: case T_MergeStmt: + case T_LetStmt: (void) test_raw_expression_coverage(parseTree, NULL); break; default: @@ -401,6 +408,11 @@ transformStmt(ParseState *pstate, Node *parseTree) (CallStmt *) parseTree); break; + case T_LetStmt: + result = transformLetStmt(pstate, + (LetStmt *) parseTree); + break; + default: /* @@ -443,6 +455,7 @@ analyze_requires_snapshot(RawStmt *parseTree) case T_MergeStmt: case T_SelectStmt: case T_PLAssignStmt: + case T_LetStmt: result = true; break; @@ -1642,6 +1655,285 @@ transformValuesClause(ParseState *pstate, SelectStmt *stmt) return qry; } +/* + * transformLetStmt - + * transform an Let Statement + */ +static Query * +transformLetStmt(ParseState *pstate, LetStmt *stmt) +{ + Query *query; + Query *result; + List *exprList = NIL; + List *exprListCoer = NIL; + ListCell *lc; + ListCell *indirection_head = NULL; + Query *selectQuery; + Oid varid; + char *attrname = NULL; + bool not_unique; + bool is_rowtype; + Oid typid; + int32 typmod; + Oid collid; + AclResult aclresult; + List *names = NULL; + int indirection_start; + int i = 0; + + /* There can't be any outer WITH to worry about */ + Assert(pstate->p_ctenamespace == NIL); + + names = NamesFromList(stmt->target); + + /* + * The AccessShareLock is created on related session variable. The lock + * will be kept for the whole transaction. + */ + varid = IdentifyVariable(names, &attrname, ¬_unique, false); + if (not_unique) + ereport(ERROR, + (errcode(ERRCODE_AMBIGUOUS_PARAMETER), + errmsg("target \"%s\" of LET command is ambiguous", + NameListToString(names)), + parser_errposition(pstate, stmt->location))); + + if (!OidIsValid(varid)) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("session variable \"%s\" doesn't exist", + NameListToString(names)), + parser_errposition(pstate, stmt->location))); + + /* + * Calculate start of possible position of an indirection in list, + * and when it is inside the list, store pointer on first node + * of indirection. + */ + indirection_start = list_length(names) - (attrname ? 1 : 0); + if (list_length(stmt->target) > indirection_start) + indirection_head = list_nth_cell(stmt->target, indirection_start); + + get_session_variable_type_typmod_collid(varid, &typid, &typmod, &collid); + + is_rowtype = type_is_rowtype(typid); + + if (attrname && !is_rowtype) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("type \"%s\" of target session variable \"%s.%s\" is not a composite type", + format_type_be(typid), + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)), + parser_errposition(pstate, stmt->location))); + + if (stmt->set_default && attrname != NULL) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("only session variable (without attribute specification) can be set to default"), + parser_errposition(pstate, stmt->location))); + + aclresult = object_aclcheck(VariableRelationId, varid, GetUserId(), ACL_UPDATE); + if (aclresult != ACLCHECK_OK) + aclcheck_error(aclresult, OBJECT_VARIABLE, NameListToString(names)); + + pstate->p_expr_kind = EXPR_KIND_LET_TARGET; + + /* + * The LET statements suppports two syntaxes: LET var = expr and LET var = + * DEFAULT. In first case, the expression is of SelectStmt node and it is + * transformated to query (SELECT) by usual way. Second syntax should be + * transformed differently. It is more cleaner do this transformation here + * (like special case), because we don't need to enhance SelectStmt about + * fields necessary for this transformation somwehere in SelectStmt + * transformation. Isn't to necessary to uglify the SelectStmt + * transformation. This is special case of LET statement, not SelectStmt. + */ + if (stmt->set_default) + { + selectQuery = makeNode(Query); + selectQuery->commandType = CMD_SELECT; + + /* + * ResTarget(SetToDefault) -> TargetEntry(expr(SetToDefault)) The + * SetToDefault is replaced by defexpr by rewriter in RewriteQuery + * function. + */ + selectQuery->targetList = transformTargetList(pstate, + ((SelectStmt *) stmt->query)->targetList, + EXPR_KIND_LET_TARGET); + } + else + selectQuery = transformStmt(pstate, stmt->query); + + /* The grammar should have produced a SELECT */ + Assert(IsA(selectQuery, Query) && selectQuery->commandType == CMD_SELECT); + + /*---------- + * Generate an expression list for the LET that selects all the + * non-resjunk columns from the subquery. + *---------- + */ + exprList = NIL; + foreach(lc, selectQuery->targetList) + { + TargetEntry *tle = (TargetEntry *) lfirst(lc); + + if (tle->resjunk) + continue; + + exprList = lappend(exprList, tle->expr); + } + + /* don't allow multicolumn result */ + if (list_length(exprList) != 1) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg_plural("assignment expression returned %d column", + "assignment expression returned %d columns", + list_length(exprList), + list_length(exprList)), + parser_errposition(pstate, + exprLocation((Node *) exprList)))); + + exprListCoer = NIL; + + foreach(lc, exprList) + { + Expr *expr = (Expr *) lfirst(lc); + Expr *coerced_expr; + Param *param; + Oid exprtypid; + + if (IsA(expr, SetToDefault)) + { + SetToDefault *def = (SetToDefault *) expr; + + def->typeId = typid; + def->typeMod = typmod; + def->collation = collid; + } + else if (IsA(expr, Const) && ((Const *) expr)->constisnull) + { + /* use known type for NULL value */ + expr = (Expr *) makeNullConst(typid, typmod, collid); + } + + /* now we can read type of expression */ + exprtypid = exprType((Node *) expr); + + param = makeNode(Param); + param->paramkind = PARAM_VARIABLE; + param->paramvarid = varid; + param->paramtype = typid; + param->paramtypmod = typmod; + + if (indirection_head) + { + bool targetIsArray; + char *targetName; + + targetName = get_session_variable_name(varid); + targetIsArray = OidIsValid(get_element_type(typid)); + + pstate->p_hasSessionVariables = true; + + coerced_expr = (Expr *) + transformAssignmentIndirection(pstate, + (Node *) param, + targetName, + targetIsArray, + typid, + typmod, + InvalidOid, + stmt->target, + indirection_head, + (Node *) expr, + COERCION_PLPGSQL, + stmt->location); + } + else + coerced_expr = (Expr *) + coerce_to_target_type(pstate, + (Node *) expr, + exprtypid, + typid, typmod, + COERCION_ASSIGNMENT, + COERCE_IMPLICIT_CAST, + stmt->location); + + if (coerced_expr == NULL) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("variable \"%s.%s\" is of type %s," + " but expression is of type %s", + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid), + format_type_be(typid), + format_type_be(exprtypid)), + errhint("You will need to rewrite or cast the expression."), + parser_errposition(pstate, exprLocation((Node *) expr)))); + + exprListCoer = lappend(exprListCoer, coerced_expr); + } + + /* + * Generate query's target list using the computed list of expressions. + */ + query = makeNode(Query); + query->commandType = CMD_SELECT; + + foreach(lc, exprListCoer) + { + Expr *expr = (Expr *) lfirst(lc); + TargetEntry *tle; + + tle = makeTargetEntry(expr, + i + 1, + FigureColname((Node *) expr), + false); + query->targetList = lappend(query->targetList, tle); + } + + /* done building the range table and jointree */ + query->rtable = pstate->p_rtable; + query->jointree = makeFromExpr(pstate->p_joinlist, NULL); + + query->hasTargetSRFs = pstate->p_hasTargetSRFs; + query->hasSubLinks = pstate->p_hasSubLinks; + query->hasSessionVariables = pstate->p_hasSessionVariables; + + /* This is top query */ + query->canSetTag = true; + + /* + * Save target session variable id. This value is used multiple times: by + * query rewriter (for getting related defexpr), by planner (for acquiring + * AccessShareLock on target variable), and by executor (we need to know + * target variable id). + */ + query->resultVariable = varid; + + assign_query_collations(pstate, query); + + stmt->query = (Node *) query; + + /* + * When statement is executed as a PlpgSQL LET statement, then we should + * return the query because we don't want to use a utilityStmt wrapper + * node. + */ + if (stmt->plpgsql_mode) + return query; + + /* represent the command as a utility Query */ + result = makeNode(Query); + result->commandType = CMD_UTILITY; + result->utilityStmt = (Node *) stmt; + + return result; +} + /* * transformSetOperationStmt - * transforms a set-operations tree diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index f5533d2472..8ad66520bb 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -306,7 +306,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); DropTransformStmt DropUserMappingStmt ExplainStmt FetchStmt GrantStmt GrantRoleStmt ImportForeignSchemaStmt IndexStmt InsertStmt - ListenStmt LoadStmt LockStmt MergeStmt NotifyStmt ExplainableStmt PreparableStmt + LetStmt ListenStmt LoadStmt LockStmt MergeStmt NotifyStmt ExplainableStmt PreparableStmt CreateFunctionStmt AlterFunctionStmt ReindexStmt RemoveAggrStmt RemoveFuncStmt RemoveOperStmt RenameStmt ReturnStmt RevokeStmt RevokeRoleStmt RuleActionStmt RuleActionStmtOrEmpty RuleStmt @@ -454,6 +454,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); TriggerTransitions TriggerReferencing vacuum_relation_list opt_vacuum_relation_list drop_option_list pub_obj_list + let_target %type <node> opt_routine_body %type <groupclause> group_clause @@ -734,7 +735,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); KEY KEYS LABEL LANGUAGE LARGE_P LAST_P LATERAL_P - LEADING LEAKPROOF LEAST LEFT LEVEL LIKE LIMIT LISTEN LOAD LOCAL + LEADING LEAKPROOF LEAST LEFT LET LEVEL LIKE LIMIT LISTEN LOAD LOCAL LOCALTIME LOCALTIMESTAMP LOCATION LOCK_P LOCKED LOGGED MAPPING MATCH MATCHED MATERIALIZED MAXVALUE MERGE METHOD @@ -1069,6 +1070,7 @@ stmt: | ImportForeignSchemaStmt | IndexStmt | InsertStmt + | LetStmt | ListenStmt | RefreshMatViewStmt | LoadStmt @@ -11966,7 +11968,8 @@ ExplainableStmt: | CreateAsStmt | CreateMatViewStmt | RefreshMatViewStmt - | ExecuteStmt /* by default all are $$=$1 */ + | ExecuteStmt + | LetStmt /* by default all are $$=$1 */ ; /***************************************************************************** @@ -11996,7 +11999,8 @@ PreparableStmt: | InsertStmt | UpdateStmt | DeleteStmt - | MergeStmt /* by default all are $$=$1 */ + | MergeStmt + | LetStmt /* by default all are $$=$1 */ ; /***************************************************************************** @@ -12584,6 +12588,49 @@ opt_hold: /* EMPTY */ { $$ = 0; } | WITHOUT HOLD { $$ = 0; } ; +/***************************************************************************** + * + * QUERY: + * LET STATEMENTS + * + *****************************************************************************/ +LetStmt: LET let_target '=' a_expr + { + LetStmt *n = makeNode(LetStmt); + SelectStmt *select; + ResTarget *res; + + n->target = $2; + + select = makeNode(SelectStmt); + res = makeNode(ResTarget); + + /* Create target list for implicit query */ + res->name = NULL; + res->indirection = NIL; + res->val = (Node *) $4; + res->location = @4; + + select->targetList = list_make1(res); + n->query = (Node *) select; + + n->set_default = IsA($4, SetToDefault); + + n->location = @2; + $$ = (Node *) n; + } + ; + +let_target: + ColId opt_indirection + { + $$ = list_make1(makeString($1)); + if ($2) + $$ = list_concat($$, + check_indirection($2, yyscanner)); + } + ; + /***************************************************************************** * * QUERY: @@ -17228,6 +17275,7 @@ unreserved_keyword: | LARGE_P | LAST_P | LEAKPROOF + | LET | LEVEL | LISTEN | LOAD @@ -17812,6 +17860,7 @@ bare_label_keyword: | LEAKPROOF | LEAST | LEFT + | LET | LEVEL | LIKE | LISTEN diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 6fc03166c6..53ff67c617 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -349,6 +349,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) Assert(false); /* can't happen */ break; case EXPR_KIND_OTHER: + case EXPR_KIND_LET_TARGET: /* * Accept aggregate/grouping here; caller must throw error if @@ -955,6 +956,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_LET_TARGET: + err = _("window functions are not allowed in LET statement"); + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_cte.c b/src/backend/parser/parse_cte.c index c5b1a49725..35aa070571 100644 --- a/src/backend/parser/parse_cte.c +++ b/src/backend/parser/parse_cte.c @@ -133,6 +133,14 @@ transformWithClause(ParseState *pstate, WithClause *withClause) errmsg("MERGE not supported in WITH query"), parser_errposition(pstate, cte->location)); + /* LET is allowed by parser, but not supported. Reject for now */ + if (IsA(cte->ctequery, LetStmt)) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("LET not supported in WITH query"), + parser_errposition(pstate, cte->location)); + + for_each_cell(rest, withClause->ctes, lnext(withClause->ctes, lc)) { CommonTableExpr *cte2 = (CommonTableExpr *) lfirst(rest); diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 03b03dc142..8889ffeb0d 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -630,6 +630,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: case EXPR_KIND_VARIABLE_DEFAULT: + case EXPR_KIND_LET_TARGET: /* okay */ break; @@ -2008,6 +2009,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_LET_TARGET: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3389,6 +3391,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_LET_TARGET: + return "LET"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index a0fdaa8605..ed5e1db03d 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2657,6 +2657,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_LET_TARGET: + err = _("set-returning functions are not allowed in LET target expression"); + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_target.c b/src/backend/parser/parse_target.c index 4cca97ff9c..022dde27f9 100644 --- a/src/backend/parser/parse_target.c +++ b/src/backend/parser/parse_target.c @@ -88,7 +88,9 @@ transformTargetEntry(ParseState *pstate, * through unmodified. (transformExpr will throw the appropriate * error if we're disallowing it.) */ - if (exprKind == EXPR_KIND_UPDATE_SOURCE && IsA(node, SetToDefault)) + if ((exprKind == EXPR_KIND_UPDATE_SOURCE || + exprKind == EXPR_KIND_LET_TARGET) + && IsA(node, SetToDefault)) expr = node; else expr = transformExpr(pstate, node, exprKind); diff --git a/src/backend/rewrite/rewriteHandler.c b/src/backend/rewrite/rewriteHandler.c index 980dc1816f..fd8ec0dbc6 100644 --- a/src/backend/rewrite/rewriteHandler.c +++ b/src/backend/rewrite/rewriteHandler.c @@ -25,6 +25,7 @@ #include "access/table.h" #include "catalog/dependency.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "commands/trigger.h" #include "executor/executor.h" #include "foreign/fdwapi.h" @@ -3701,6 +3702,39 @@ RewriteQuery(Query *parsetree, List *rewrite_events, int orig_rt_length) } } + /* + * Rewrite SetToDefault by default expression of Let statement. + */ + if (event == CMD_SELECT && OidIsValid(parsetree->resultVariable)) + { + Oid resultVariable = parsetree->resultVariable; + + if (list_length(parsetree->targetList) == 1 && + IsA(((TargetEntry *) linitial(parsetree->targetList))->expr, SetToDefault)) + { + Variable var; + TargetEntry *tle; + TargetEntry *newtle; + Expr *defexpr; + + /* Read session variable metadata with defexpr */ + InitVariable(&var, resultVariable, false); + + if (var.has_defexpr) + defexpr = (Expr *) var.defexpr; + else + defexpr = (Expr *) makeNullConst(var.typid, var.typmod, var.collation); + + tle = (TargetEntry *) linitial(parsetree->targetList); + newtle = makeTargetEntry(defexpr, + tle->resno, + pstrdup(tle->resname), + false); + + parsetree->targetList = list_make1(newtle); + } + } + /* * If the statement is an insert, update, delete, or merge, adjust its * targetlist as needed, and then fire INSERT/UPDATE/DELETE rules on it. diff --git a/src/backend/tcop/dest.c b/src/backend/tcop/dest.c index c0406e2ee5..86dbf370ac 100644 --- a/src/backend/tcop/dest.c +++ b/src/backend/tcop/dest.c @@ -37,6 +37,7 @@ #include "executor/functions.h" #include "executor/tqueue.h" #include "executor/tstoreReceiver.h" +#include "executor/svariableReceiver.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "utils/portal.h" @@ -152,6 +153,9 @@ CreateDestReceiver(CommandDest dest) case DestTupleQueue: return CreateTupleQueueDestReceiver(NULL); + + case DestVariable: + return CreateVariableDestReceiver(); } /* should never get here */ @@ -187,6 +191,7 @@ EndCommand(const QueryCompletion *qc, CommandDest dest, bool force_undecorated_o case DestSQLFunction: case DestTransientRel: case DestTupleQueue: + case DestVariable: break; } } @@ -232,6 +237,7 @@ NullCommand(CommandDest dest) case DestSQLFunction: case DestTransientRel: case DestTupleQueue: + case DestVariable: break; } } @@ -275,6 +281,7 @@ ReadyForQuery(CommandDest dest) case DestSQLFunction: case DestTransientRel: case DestTupleQueue: + case DestVariable: break; } } diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index ddc26794b2..40cc8231fb 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -242,6 +242,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree) case T_CallStmt: case T_DoStmt: + case T_LetStmt: { /* * Commands inside the DO block or the called procedure might @@ -1074,6 +1075,11 @@ standard_ProcessUtility(PlannedStmt *pstmt, break; } + case T_LetStmt: + ExecuteLetStmt(pstate, (LetStmt *) parsetree, params, + queryEnv, qc); + break; + default: /* All other statement types have event trigger support */ ProcessUtilitySlow(pstate, pstmt, queryString, @@ -2207,6 +2213,10 @@ UtilityContainsQuery(Node *parsetree) return UtilityContainsQuery(qry->utilityStmt); return qry; + case T_LetStmt: + qry = castNode(Query, ((LetStmt *) parsetree)->query); + return qry; + default: return NULL; } @@ -2405,6 +2415,10 @@ CreateCommandTag(Node *parsetree) tag = CMDTAG_SELECT; break; + case T_LetStmt: + tag = CMDTAG_LET; + break; + /* utility statements --- same whether raw or cooked */ case T_TransactionStmt: { @@ -3290,6 +3304,7 @@ GetCommandLogLevel(Node *parsetree) break; case T_PLAssignStmt: + case T_LetStmt: lev = LOGSTMT_ALL; break; diff --git a/src/backend/utils/cache/plancache.c b/src/backend/utils/cache/plancache.c index a3bea4e951..d03fe5ec4f 100644 --- a/src/backend/utils/cache/plancache.c +++ b/src/backend/utils/cache/plancache.c @@ -135,6 +135,7 @@ InitPlanCache(void) CacheRegisterSyscacheCallback(AMOPOPID, PlanCacheSysCallback, (Datum) 0); CacheRegisterSyscacheCallback(FOREIGNSERVEROID, PlanCacheSysCallback, (Datum) 0); CacheRegisterSyscacheCallback(FOREIGNDATAWRAPPEROID, PlanCacheSysCallback, (Datum) 0); + CacheRegisterSyscacheCallback(VARIABLEOID, PlanCacheObjectCallback, (Datum) 0); } /* @@ -1886,6 +1887,17 @@ ScanQueryForLocks(Query *parsetree, bool acquire) (void *) &acquire, QTW_IGNORE_RC_SUBQUERIES); } + + /* Process session variables */ + if (OidIsValid(parsetree->resultVariable)) + { + if (acquire) + LockDatabaseObject(VariableRelationId, parsetree->resultVariable, + 0, AccessShareLock); + else + UnlockDatabaseObject(VariableRelationId, parsetree->resultVariable, + 0, AccessShareLock); + } } /* diff --git a/src/include/commands/session_variable.h b/src/include/commands/session_variable.h index f442c5d3b8..ceb0d357da 100644 --- a/src/include/commands/session_variable.h +++ b/src/include/commands/session_variable.h @@ -39,4 +39,7 @@ extern void AtEOSubXact_SessionVariable(bool isCommit, SubTransactionId mySubid, SubTransactionId parentSubid); +extern void ExecuteLetStmt(ParseState *pstate, LetStmt *stmt, ParamListInfo params, + QueryEnvironment *queryEnv, QueryCompletion *qc); + #endif diff --git a/src/include/executor/svariableReceiver.h b/src/include/executor/svariableReceiver.h new file mode 100644 index 0000000000..63f6ee9b7f --- /dev/null +++ b/src/include/executor/svariableReceiver.h @@ -0,0 +1,25 @@ +/*------------------------------------------------------------------------- + * + * svariableReceiver.h + * prototypes for svariableReceiver.c + * + * + * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/executor/svariableReceiver.h + * + *------------------------------------------------------------------------- + */ + +#ifndef SVARIABLE_RECEIVER_H +#define SVARIABLE_RECEIVER_H + +#include "tcop/dest.h" + + +extern DestReceiver *CreateVariableDestReceiver(void); + +extern void SetVariableDestReceiverVarid(DestReceiver *self, Oid varid); + +#endif /* SVARIABLE_RECEIVER_H */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 1f067fc84f..b96b56fd6a 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -149,6 +149,9 @@ typedef struct Query */ int resultRelation pg_node_attr(query_jumble_ignore); + /* target variable of LET statement */ + Oid resultVariable; + /* has aggregates in tlist or havingQual */ bool hasAggs pg_node_attr(query_jumble_ignore); /* has window functions in tlist */ @@ -1918,6 +1921,21 @@ typedef struct MergeStmt WithClause *withClause; /* WITH clause */ } MergeStmt; +/* ---------------------- + * Let Statement + * ---------------------- + */ +typedef struct LetStmt +{ + NodeTag type; + List *target; /* target variable */ + Node *query; /* source expression */ + bool set_default; /* true, when set to DEFAULt is wanted */ + bool plpgsql_mode; /* true, when command will be executed + * (parsed) by plpgsql runtime */ + int location; +} LetStmt; + /* ---------------------- * Select Statement * diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index d287a80dce..69084136e7 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -50,7 +50,7 @@ typedef struct PlannedStmt NodeTag type; - CmdType commandType; /* select|insert|update|delete|merge|utility */ + CmdType commandType; /* select|let|insert|update|delete|merge|utility */ uint64 queryId; /* query identifier (copied from Query) */ diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 9f79b90251..7f2445c049 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -246,6 +246,7 @@ PG_KEYWORD("leading", LEADING, RESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("leakproof", LEAKPROOF, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("least", LEAST, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("left", LEFT, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) +PG_KEYWORD("let", LET, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("level", LEVEL, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("like", LIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("limit", LIMIT, RESERVED_KEYWORD, AS_LABEL) diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h index 4e9692d05a..c3f1ccb4c0 100644 --- a/src/include/parser/parse_node.h +++ b/src/include/parser/parse_node.h @@ -82,6 +82,7 @@ typedef enum ParseExprKind EXPR_KIND_GENERATED_COLUMN, /* generation expression for a column */ EXPR_KIND_CYCLE_MARK, /* cycle mark value */ EXPR_KIND_VARIABLE_DEFAULT, /* default value for session variable */ + EXPR_KIND_LET_TARGET, /* LET assignment (should be same like UPDATE) */ } ParseExprKind; diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h index 259bdc994e..648a4af305 100644 --- a/src/include/tcop/cmdtaglist.h +++ b/src/include/tcop/cmdtaglist.h @@ -186,6 +186,7 @@ PG_CMDTAG(CMDTAG_GRANT, "GRANT", true, false, false) PG_CMDTAG(CMDTAG_GRANT_ROLE, "GRANT ROLE", false, false, false) PG_CMDTAG(CMDTAG_IMPORT_FOREIGN_SCHEMA, "IMPORT FOREIGN SCHEMA", true, false, false) PG_CMDTAG(CMDTAG_INSERT, "INSERT", false, false, true) +PG_CMDTAG(CMDTAG_LET, "LET", false, false, false) PG_CMDTAG(CMDTAG_LISTEN, "LISTEN", false, false, false) PG_CMDTAG(CMDTAG_LOAD, "LOAD", false, false, false) PG_CMDTAG(CMDTAG_LOCK_TABLE, "LOCK TABLE", false, false, false) diff --git a/src/include/tcop/dest.h b/src/include/tcop/dest.h index a7d86e7abd..e7dd749949 100644 --- a/src/include/tcop/dest.h +++ b/src/include/tcop/dest.h @@ -95,7 +95,8 @@ typedef enum DestCopyOut, /* results sent to COPY TO code */ DestSQLFunction, /* results sent to SQL-language func mgr */ DestTransientRel, /* results sent to transient relation */ - DestTupleQueue /* results sent to tuple queue */ + DestTupleQueue, /* results sent to tuple queue */ + DestVariable /* results sents to session variable */ } CommandDest; /* ---------------- diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 69671f2c78..5053ab081e 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1407,6 +1407,7 @@ LargeObjectDesc LastAttnumInfo Latch LerpFunc +LetStmt LexDescr LexemeEntry LexemeHashKey @@ -2668,6 +2669,7 @@ SupportRequestRows SupportRequestSelectivity SupportRequestSimplify SupportRequestWFuncMonotonic +SVariableState SVariable SVariableData SVariableXActAction -- 2.40.1 [text/x-patch] v20230517-1-0002-session-variables.patch (111.8K, ../../CAFj8pRAM60yANCnqKHHXZM3j++wpXWOu+rw6tVq5m531LHXiSw@mail.gmail.com/11-v20230517-1-0002-session-variables.patch) download | inline diff: From 2c998eb265293450c37f190eb16bb113bb5ae4fa Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Sun, 13 Nov 2022 17:45:23 +0100 Subject: [PATCH 02/10] session variables Implementation storage and access routines. Session variables are stored in session memory inside dedicated hash table. Two levels of an access are implemented: API level and SQL level. Both levels are implemented in this commit. The most difficult part is cleaning (reset) of session variable. The content of session variable should be cleaned when related session variable is removed from system catalog. But queue of sinval messages can be truncated, or current transaction state can disallow an access to system catalog, so cleaning can be postponed. --- src/backend/access/transam/xact.c | 2 +- src/backend/catalog/dependency.c | 5 + src/backend/catalog/namespace.c | 327 ++++++ src/backend/commands/meson.build | 1 + src/backend/commands/session_variable.c | 1235 +++++++++++++++++++-- src/backend/executor/execExpr.c | 76 ++ src/backend/executor/execExprInterp.c | 17 + src/backend/executor/execMain.c | 58 + src/backend/executor/execParallel.c | 147 ++- src/backend/jit/llvm/llvmjit_expr.c | 6 + src/backend/optimizer/plan/planner.c | 8 + src/backend/optimizer/plan/setrefs.c | 118 +- src/backend/optimizer/prep/prepjointree.c | 3 + src/backend/optimizer/util/clauses.c | 74 +- src/backend/parser/analyze.c | 7 + src/backend/parser/parse_expr.c | 253 ++++- src/backend/tcop/pquery.c | 3 + src/backend/utils/adt/ruleutils.c | 46 + src/backend/utils/cache/plancache.c | 29 +- src/backend/utils/fmgr/fmgr.c | 10 +- src/backend/utils/misc/guc_tables.c | 10 + src/include/catalog/namespace.h | 1 + src/include/catalog/pg_proc.dat | 9 +- src/include/commands/session_variable.h | 12 +- src/include/executor/execExpr.h | 10 + src/include/executor/execdesc.h | 4 + src/include/nodes/execnodes.h | 19 + src/include/nodes/parsenodes.h | 2 + src/include/nodes/pathnodes.h | 5 + src/include/nodes/plannodes.h | 2 + src/include/nodes/primnodes.h | 12 +- src/include/optimizer/planmain.h | 2 + src/include/parser/parse_expr.h | 1 + src/include/parser/parse_node.h | 1 + src/tools/pgindent/typedefs.list | 3 + 35 files changed, 2416 insertions(+), 102 deletions(-) diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index ee20c2e520..0ccdb56ff4 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -2225,7 +2225,7 @@ CommitTransaction(void) */ smgrDoPendingSyncs(true, is_parallel_worker); - /* Let ON COMMIT DROP */ + /* Let ON COMMIT DROP or ON TRANSACTION END */ AtPreEOXact_SessionVariable(true); /* close large objects before lower-level cleanup */ diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index 77acbeda80..cae1c7c1f0 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -1915,6 +1915,11 @@ find_expr_references_walker(Node *node, { Param *param = (Param *) node; + /* A variable parameter depends on the session variable */ + if (param->paramkind == PARAM_VARIABLE) + add_object_address(OCLASS_VARIABLE, param->paramvarid, 0, + context->addrs); + /* A parameter must depend on the parameter's datatype */ add_object_address(OCLASS_TYPE, param->paramtype, 0, context->addrs); diff --git a/src/backend/catalog/namespace.c b/src/backend/catalog/namespace.c index 9a04530d1e..73d7996160 100644 --- a/src/backend/catalog/namespace.c +++ b/src/backend/catalog/namespace.c @@ -2970,6 +2970,333 @@ LookupVariable(const char *nspname, return varoid; } +/* + * The input list contains names with indirection expressions used as the left + * part of LET statement. The following routine returns a new list with only + * initial strings (names) - without indirection expressions. + */ +List * +NamesFromList(List *names) +{ + ListCell *l; + List *result = NIL; + + foreach(l, names) + { + Node *n = lfirst(l); + + if (IsA(n, String)) + { + result = lappend(result, n); + } + else + break; + } + + return result; +} + +/* + * IdentifyVariable - try to find variable identified by list of names. + * + * Before this call we don't know, how these fields should be mapped to + * schema name, variable name and attribute name. In this routine + * we try to apply passed names to all possible combinations of schema name, + * variable name and attribute name, and we count valid combinations. + * + * Returns oid of identified variable. When last field of names list is + * identified as an attribute, then output attrname argument is set to + * an string of this field. + * + * When there is not any valid combination, then we are sure, so the + * list of names cannot to identify any session variable. In this case + * we return InvalidOid. + * + * We can find more valid combination than one. + * Example: users can have session variable x in schema y, and + * session variable y with attribute x inside some schema from + * search path. In this situation the meaning of expression "y"."x" + * is ambiguous. In this case this routine returns oid of variable + * x in schema y, and the output parameter "not_unique" is set to + * true. In this case this variable is locked. + * + * The AccessShareLock is created on related session variable. The lock + * will be kept for the whole transaction. + * + * Note: the out attrname should be used only when the session variable + * is identified. When the session variable is not identified, then this + * output variable can hold reference to some string, but isn't sure + * about its semantics. + * + * When we use this routine for identification of shadowed variable, + * we don't want to raise any error. Shadowing column reference is correct, + * and we don't want to break execution due shadowing check. + */ +Oid +IdentifyVariable(List *names, char **attrname, bool *not_unique, bool noerror) +{ + Node *field1 = NULL; + Node *field2 = NULL; + Node *field3 = NULL; + Node *field4 = NULL; + char *a = NULL; + char *b = NULL; + char *c = NULL; + char *d = NULL; + Oid varid = InvalidOid; + Oid old_varid = InvalidOid; + uint64 inval_count; + bool retry = false; + + /* + * DDL operations can change the results of a name lookup. Since all such + * operations will generate invalidation messages, we keep track of + * whether any such messages show up while we're performing the operation, + * and retry until either (1) no more invalidation messages show up or (2) + * the answer doesn't change. + */ + for (;;) + { + Oid varoid_without_attr = InvalidOid; + Oid varoid_with_attr = InvalidOid; + + *not_unique = false; + *attrname = NULL; + varid = InvalidOid; + + inval_count = SharedInvalidMessageCounter; + + switch (list_length(names)) + { + case 1: + field1 = linitial(names); + + Assert(IsA(field1, String)); + + varid = LookupVariable(NULL, strVal(field1), true); + break; + + case 2: + field1 = linitial(names); + field2 = lsecond(names); + + Assert(IsA(field1, String)); + a = strVal(field1); + + if (IsA(field2, String)) + { + /* when both fields are of string type */ + b = strVal(field2); + + /* + * a.b can mean "schema"."variable" or "variable"."field". + * Check both variants, and returns InvalidOid with + * not_unique flag, when both interpretations are + * possible. + */ + varoid_without_attr = LookupVariable(a, b, true); + varoid_with_attr = LookupVariable(NULL, a, true); + } + else + { + /* The last field of list can be star too. */ + Assert(IsA(field2, A_Star)); + + /* + * In this case, the field1 should be variable name. + * But direct unboxing of composite session variables + * is not supported now, and then we don't need to try + * lookup related variable. + * + * Unboxing is supported by syntax (var).* + */ + return InvalidOid; + } + + if (OidIsValid(varoid_without_attr) && OidIsValid(varoid_with_attr)) + { + *not_unique = true; + varid = varoid_without_attr; + } + else if (OidIsValid(varoid_without_attr)) + { + varid = varoid_without_attr; + } + else if (OidIsValid(varoid_with_attr)) + { + *attrname = b; + varid = varoid_with_attr; + } + break; + + case 3: + { + bool field1_is_catalog = false; + + field1 = linitial(names); + field2 = lsecond(names); + field3 = lthird(names); + + Assert(IsA(field1, String)); + Assert(IsA(field2, String)); + + a = strVal(field1); + b = strVal(field2); + + if (IsA(field3, String)) + { + c = strVal(field3); + + /* + * a.b.c can mean catalog.schema.variable + * or schema.variable.field. + * + * Check both variants, and set not_unique flag, + * when both interpretations are possible. + * + * When third node is star, only possible + * interpretation is schema.variable.*, but this + * pattern is not supported now. + */ + varoid_with_attr = LookupVariable(a, b, true); + + /* + * check pattern catalog.schema.variable only when + * there is possibility to success. + */ + if (strcmp(a, get_database_name(MyDatabaseId)) == 0) + { + field1_is_catalog = true; + varoid_without_attr = LookupVariable(b, c, true); + } + } + else + { + Assert(IsA(field3, A_Star)); + return InvalidOid; + } + + if (OidIsValid(varoid_without_attr) && OidIsValid(varoid_with_attr)) + { + *not_unique = true; + varid = varoid_without_attr; + } + else if (OidIsValid(varoid_without_attr)) + { + varid = varoid_without_attr; + } + else if (OidIsValid(varoid_with_attr)) + { + *attrname = c; + varid = varoid_with_attr; + } + + /* + * When we didn't find variable, we can (when it is allowed) + * raise cross-database reference error. + */ + if (!OidIsValid(varid) && !noerror && !field1_is_catalog) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cross-database references are not implemented: %s", + NameListToString(names)))); + } + break; + + case 4: + { + field1 = linitial(names); + field2 = lsecond(names); + field3 = lthird(names); + field4 = lfourth(names); + + Assert(IsA(field1, String)); + Assert(IsA(field2, String)); + Assert(IsA(field3, String)); + + a = strVal(field1); + b = strVal(field2); + c = strVal(field3); + + /* + * In this case, "a" is used as catalog name - check it. + */ + if (strcmp(a, get_database_name(MyDatabaseId)) != 0) + { + if (!noerror) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cross-database references are not implemented: %s", + NameListToString(names)))); + } + else + { + if (IsA(field4, String)) + { + d = strVal(field4); + } + else + { + Assert(IsA(field4, A_Star)); + return InvalidOid; + } + + *attrname = d; + varid = LookupVariable(b, c, true); + } + } + break; + + default: + if (!noerror) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("improper qualified name (too many dotted names): %s", + NameListToString(names)))); + return InvalidOid; + } + + /* + * If, upon retry, we get back the same OID we did last time, then the + * invalidation messages we processed did not change the final answer. + * So we're done. + * + * If we got a different OID, we've locked the variable that used to + * have this name rather than the one that does now. So release the + * lock. + */ + if (retry) + { + if (old_varid == varid) + break; + + if (OidIsValid(old_varid)) + UnlockDatabaseObject(VariableRelationId, old_varid, 0, AccessShareLock); + } + + /* + * Lock the variable. This will also accept any pending invalidation + * messages. If we got back InvalidOid, indicating not found, then + * there's nothing to lock, but we accept invalidation messages + * anyway, to flush any negative catcache entries that may be + * lingering. + */ + if (!OidIsValid(varid)) + AcceptInvalidationMessages(); + else if (OidIsValid(varid)) + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); + + if (inval_count == SharedInvalidMessageCounter) + break; + + retry = true; + old_varid = varid; + varid = InvalidOid; + } + + return varid; +} + /* * DeconstructQualifiedName * Given a possibly-qualified name expressed as a list of String nodes, diff --git a/src/backend/commands/meson.build b/src/backend/commands/meson.build index 42cced9ebe..404be3d22b 100644 --- a/src/backend/commands/meson.build +++ b/src/backend/commands/meson.build @@ -38,6 +38,7 @@ backend_sources += files( 'schemacmds.c', 'seclabel.c', 'sequence.c', + 'session_variable.c', 'statscmds.c', 'subscriptioncmds.c', 'tablecmds.c', diff --git a/src/backend/commands/session_variable.c b/src/backend/commands/session_variable.c index 946e73e467..b4a91ea5a3 100644 --- a/src/backend/commands/session_variable.c +++ b/src/backend/commands/session_variable.c @@ -8,7 +8,7 @@ * * * IDENTIFICATION - * src/backend/commands/sessionvariable.c + * src/backend/commands/session_variable.c * *------------------------------------------------------------------------- */ @@ -22,87 +22,244 @@ #include "commands/session_variable.h" #include "funcapi.h" #include "miscadmin.h" +#include "optimizer/optimizer.h" +#include "storage/lmgr.h" +#include "storage/proc.h" #include "utils/builtins.h" +#include "utils/datum.h" +#include "utils/inval.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/syscache.h" /* - * The life cycle of temporary session variable can be - * limmited by using clause ON COMMIT DROP. + * Values of session variables are stored in the backend local memory, + * in sessionvars hash table in binary format, in a dedicated memory + * context SVariableMemoryContext. A session variable value can stay + * valid for longer than the transaction that assigns its value. To + * make sure that the underlying memory is eventually freed, but not + * before it's guarantee that the value won't be needed anymore, we + * need to handle the two following points: + * + * - We need detect when a variable is dropped, whether in the current + * transaction in the current session or by another session, and mark + * the underlying entries for removal. To protect the content against + * possibly rollbacked DROP VARIABLE commands, the entries (and + * memory) shouldn't be freed immediately but be postponed until the + * end of the transaction. + * + * - The session variable can be dropped explicitly (by DROP VARIABLE + * command) or implicitly (using ON COMMIT DROP clause), and the + * value can be implicitly removed (using the ON TRANSACTION END + * clause). In all those cases the memory should also be freed at + * the transaction end. + * + * To achieve that, we maintain 3 queues of actions to be performed at + * certain time: + * - a List of SVariableXActActionItem, to handle ON COMMIT DROP + * variables, and delayed memory cleanup of variable dropped by the + * current transaction. Those actions are transactional (for instance + * we don't want to cleanup the memory of a rollbacked DROP VARIABLE) + * so the structure is needed to keep track of the final transaction + * state + * - a List of variable Oid for the ON TRANSACTION ON RESET variables + * - a List of variable Oid for the concurrent DROP VARIABLE + * notification we receive via shared invalidations. + * + * Note that although resetting a variable doesn't technically require + * to remove the entry from the sessionvars hash table, we currently + * do it. It's a simple way to implement the reset, and helps to + * reduce memory usage and prevents the hash table from bloating. + * + * There are two different ways to do the final access to session + * variables: buffered (indirect) or direct. Buffered access is used + * in regular DML statements, where we have to ensure the stability of + * the variable values. The session variables have the same behaviour + * as external query parameters, which is consistent with using + * PL/pgSQL's variables in embedded queries in PL/pgSQL. + * + * This is implemented by using an aux buffer (an array) that holds a + * copy of values of used (in query) session variables, which is also + * transmitted to the parallel workers. The values from this array + * are passed as constant (EEOP_CONST). + * + * Direct access is used by simple expression evaluation (PLpgSQL). + * In this case we don't need to ensure the stability of passed + * values, and maintaining the buffer with copies of values of session + * variables would be useless overhead. In this case we just read the + * value of the session variable directly (EEOP_PARAM_VARIABLE). This + * strategy removes the necessity to modify related PL/pgSQL code to + * support session variables (the reading of session variables is + * fully transparent for PL/pgSQL). */ typedef enum SVariableXActAction { SVAR_ON_COMMIT_DROP, /* used for ON COMMIT DROP */ + SVAR_ON_COMMIT_RESET, /* used for DROP VARIABLE */ } SVariableXActAction; typedef struct SVariableXActActionItem { Oid varid; /* varid of session variable */ + SVariableXActAction action; /* - * creating_subid is the ID of the creating subxact. If the action was - * unregistered during the current transaction, deleting_subid is the ID - * of the deleting subxact, otherwise InvalidSubTransactionId. + * creating_subid is the ID of the sub-transaction that registered + * the action. If the action was unregistered during the current + * transaction, deleting_subid is the ID of the deleting + * sub-transaction, otherwise InvalidSubTransactionId. */ SubTransactionId creating_subid; SubTransactionId deleting_subid; } SVariableXActActionItem; -/* List holds fields of SVariableXActActionItem type */ -static List *xact_drop_actions = NIL; - -static void register_session_variable_xact_action(Oid varid, SVariableXActAction action); -static void unregister_session_variable_xact_action(Oid varid, SVariableXActAction action); +/* List of SVariableXActActionItem */ +static List *xact_on_commit_actions = NIL; +/* + * To process ON TRANSACTION END RESET variables, for which we always + * need to clear the saved values. + */ +static List *xact_reset_varids = NIL; /* - * Do the necessary work to setup local memory management of a new - * variable. - * - * Caller should already have created the necessary entry in catalog - * and made them visible. + * When the session variable is dropped we need to free local memory. The + * session variable can be dropped by current session, but it can be + * dropped by other's sessions too, so we have to watch sinval message. + * But because we don't want to free local memory immediately, we need to + * hold list of possibly dropped session variables and at the end of + * transaction, we check session variables from this list against system + * catalog. This check can be postponed into next transaction if + * current transactions is in aborted state, as we wouldn't be able to + * access the system catalog. */ -void -SessionVariableCreatePostprocess(Oid varid, char eoxaction) +static List *xact_recheck_varids = NIL; + +typedef struct SVariableData { + Oid varid; /* pg_variable OID of the variable (hash key) */ + /* - * For temporary variables, we need to create a new end of xact action to - * ensure deletion from catalog. + * The session variable is identified by oid. The oid is unique in + * catalog. Unfortunately, the memory cleanup can be postponed to + * the beginning of the session next transaction, and it's possible + * that this next transaction sees a different variable with the same oid. + * We therefore need an extra identifier to distinguish both cases. We + * use the LSN number of session variable at creation time. The + * value of session variable (in memory) is valid, when there is a + * record in pg_variable with same oid and same create_lsn. */ - if (eoxaction == VARIABLE_EOX_DROP) - { - Assert(isTempNamespace(get_session_variable_namespace(varid))); + XLogRecPtr create_lsn; - register_session_variable_xact_action(varid, SVAR_ON_COMMIT_DROP); + bool isnull; + bool freeval; + Datum value; + + Oid typid; + int16 typlen; + bool typbyval; + + bool is_domain; + void *domain_check_extra; + LocalTransactionId domain_check_extra_lxid; + + /* + * Top level local transaction id of the last transaction that dropped the + * variable if any. We need this information to avoid freeing memory for + * variable dropped by the local backend that may be eventually rollbacked. + */ + LocalTransactionId drop_lxid; + + bool is_not_null; /* don't allow null values */ + bool is_immutable; /* true when variable is immutable */ + bool has_defexpr; /* true when variable has a default value */ + + bool is_valid; /* true when variable was successfully + * initialized */ + + uint32 hashvalue; /* used for pairing sinval message */ + + bool eox_reset; /* true, when lifecycle is limitted by + * transaction */ +} SVariableData; + +typedef SVariableData *SVariable; + +static HTAB *sessionvars = NULL; /* hash table for session variables */ + +static MemoryContext SVariableMemoryContext = NULL; + +static void create_sessionvars_hashtables(void); +static void free_session_variable_value(SVariable svar); +static void init_session_variable(SVariable svar, Variable *var); +static bool is_session_variable_valid(SVariable svar); +static void pg_variable_cache_callback(Datum arg, int cacheid, + uint32 hashvalue); +static SVariable prepare_variable_for_reading(Oid varid); +static void register_session_variable_xact_action(Oid varid, + SVariableXActAction action); +static void remove_session_variable(SVariable svar); +static void remove_session_variable_by_id(Oid varid); +static void set_session_variable(SVariable svar, Datum value, bool isnull, + bool init_mode); +static const char *SVariableXActActionName(SVariableXActAction action); +static void sync_sessionvars_all(bool filter_lxid); +static void unregister_session_variable_xact_action(Oid varid, + SVariableXActAction action); + + +/* + * Returns human readable name of SVariableXActAction value. + */ +static const char * +SVariableXActActionName(SVariableXActAction action) +{ + switch (action) + { + case SVAR_ON_COMMIT_DROP: + return "ON COMMIT DROP"; + case SVAR_ON_COMMIT_RESET: + return "ON COMMIT RESET"; + default: + elog(ERROR, "unknown SVariableXActAction action %d", action); } } /* - * Handle the local memory cleanup for a DROP VARIABLE command. - * - * Caller should take care of removing the pg_variable entry first. + * Free all memory allocated for the given session variable, but + * preserve the hash entry in sessionvars. */ -void -SessionVariableDropPostprocess(Oid varid) +static void +free_session_variable_value(SVariable svar) { + /* Clean current value */ + if (!svar->isnull) + { + if (svar->freeval) + { + pfree(DatumGetPointer(svar->value)); + svar->freeval = false; + } + + svar->isnull = true; + } + + svar->value = (Datum) 0; + svar->freeval = false; + /* - * The entry was removed from catalog already, we must not do it - * again at end of xact time. + * We can mark this session variable as valid when it has not default + * expression, and when null is allowed. When it has defexpr, then the + * content will be valid after an assignment or defexp evaluation. */ - unregister_session_variable_xact_action(varid, SVAR_ON_COMMIT_DROP); + svar->is_valid = !svar->has_defexpr && !svar->is_not_null; } /* * Registration of actions to be executed on session variables at transaction * end time. We want to drop temporary session variables with clause ON COMMIT - * DROP, or we want to reset values of session variables with clause ON - * TRANSACTION END RESET or we want to clean (reset) local memory allocated by - * values of dropped session variables. - */ - -/* - * Register a session variable xact action. + * DROP, or we want to clean (reset) local memory allocated by + * values of session variables dropped in other backends. */ static void register_session_variable_xact_action(Oid varid, @@ -111,27 +268,30 @@ register_session_variable_xact_action(Oid varid, SVariableXActActionItem *xact_ai; MemoryContext oldcxt; - oldcxt = MemoryContextSwitchTo(CacheMemoryContext); + elog(DEBUG1, "SVariableXActAction \"%s\" is registered for session variable (oid:%u)", + SVariableXActActionName(action), varid); + + oldcxt = MemoryContextSwitchTo(TopTransactionContext); xact_ai = (SVariableXActActionItem *) palloc(sizeof(SVariableXActActionItem)); xact_ai->varid = varid; + xact_ai->action = action; xact_ai->creating_subid = GetCurrentSubTransactionId(); xact_ai->deleting_subid = InvalidSubTransactionId; - Assert(action == SVAR_ON_COMMIT_DROP); - xact_drop_actions = lcons(xact_ai, xact_drop_actions); + xact_on_commit_actions = lcons(xact_ai, xact_on_commit_actions); MemoryContextSwitchTo(oldcxt); } /* - * Unregister an action on a given session variable from action list. In this - * moment, the action is just marked as deleted by setting deleting_subid. The - * calling even might be rollbacked, in which case we should not lose this - * action. + * Unregister an action on a given session variable from the action list. + * The action is just marked as deleted by setting deleting_subid. + * The calling subtransaction even might be rollbacked, in which case the + * action shouldn't be removed. */ static void unregister_session_variable_xact_action(Oid varid, @@ -139,43 +299,842 @@ unregister_session_variable_xact_action(Oid varid, { ListCell *l; - Assert(action == SVAR_ON_COMMIT_DROP); + elog(DEBUG1, "SVariableXActAction \"%s\" is unregistered for session variable (oid:%u)", + SVariableXActActionName(action), varid); - foreach(l, xact_drop_actions) + foreach(l, xact_on_commit_actions) { SVariableXActActionItem *xact_ai = (SVariableXActActionItem *) lfirst(l); - if (xact_ai->varid == varid) + if (xact_ai->action == action && xact_ai->varid == varid) xact_ai->deleting_subid = GetCurrentSubTransactionId(); } } /* - * Perform ON TRANSACTION END RESET or ON COMMIT DROP - * and COMMIT/ROLLBACK of transaction session variables. + * Release the given session variable from sessionvars hashtab and free + * all underlying allocated memory. + */ +static void +remove_session_variable(SVariable svar) +{ + free_session_variable_value(svar); + + /* + * In this moment, the session variable is not in catalog, so only saved + * oid can be displayed. + */ + elog(DEBUG1, "session variable (oid:%u) is removing from memory", + svar->varid); + + if (hash_search(sessionvars, + (void *) &svar->varid, + HASH_REMOVE, + NULL) == NULL) + elog(DEBUG1, "hash table corrupted"); +} + +/* + * Release the session variable defined by varid from sessionvars + * hashtab and free all underlying allocated memory. + */ +static void +remove_session_variable_by_id(Oid varid) +{ + SVariable svar; + bool found; + + if (!sessionvars) + return; + + svar = (SVariable) hash_search(sessionvars, &varid, + HASH_FIND, &found); + if (found) + remove_session_variable(svar); +} + +/* + * Callback function for session variable invalidation. + * + * It queues a list of variable Oid in xact_recheck_varids. + */ +static void +pg_variable_cache_callback(Datum arg, int cacheid, uint32 hashvalue) +{ + HASH_SEQ_STATUS status; + SVariable svar; + + /* + * There is no guarantee of sessionvars being initialized, even when + * receiving an invalidation callback, as DISCARD [ ALL | VARIABLES ] + * destroys the hash table entirely. + */ + if (!sessionvars) + return; + + elog(DEBUG1, "pg_variable_cache_callback %u %u", cacheid, hashvalue); + + /* + * When the hashvalue is not specified, then we have to recheck all + * currently used session variables. Since we can't guarantee the exact + * session variable from its hashValue, we also have to iterate over + * all items of the sessionvars hash table. + */ + hash_seq_init(&status, sessionvars); + + while ((svar = (SVariable) hash_seq_search(&status)) != NULL) + { + if (hashvalue == 0 || svar->hashvalue == hashvalue) + { + MemoryContext oldcxt; + + /* The list needs to be able to survive the transaction */ + oldcxt = MemoryContextSwitchTo(SVariableMemoryContext); + + xact_recheck_varids = lappend_oid(xact_recheck_varids, + svar->varid); + + MemoryContextSwitchTo(oldcxt); + + elog(DEBUG1, "session variable (oid:%u) should be rechecked (forced by sinval)", + svar->varid); + } + + /* + * although it there is low probability, we have to iterate over all + * locally set session variables, because hashvalue is not a unique + * identifier. + */ + } +} + +/* + * Returns true when the entry in pg_variable is valid for the given session + * variable. + */ +static bool +is_session_variable_valid(SVariable svar) +{ + HeapTuple tp; + bool result = false; + + tp = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(svar->varid)); + + if (HeapTupleIsValid(tp)) + { + /* + * In this case, the only oid cannot be used as unique identifier, + * because the oid counter can wraparound, and the oid can be used for + * new other session variable. We do a second check against 64bit + * unique identifier. + */ + if (svar->create_lsn == ((Form_pg_variable) GETSTRUCT(tp))->varcreate_lsn) + result = true; + + ReleaseSysCache(tp); + } + + return result; +} + +/* + * Recheck the possibly invalidated variables (in memory) against system + * catalog. This routine is called before any read or any write from/to session + * variables and when processing a committed transaction. + * If filter_lxid is true, this function will ignore the recheck for variables + * that have the same cached local transaction id as the transaction current + * top level local transaction id, ie. the variables dropped in the current top + * level transaction or any underlying subtransaction. + */ +static void +sync_sessionvars_all(bool filter_lxid) +{ + SVariable svar; + ListCell *l; + List *xact_recheck_varids_snapshot; + + if (!xact_recheck_varids) + return; + + /* + * If the sessionvars hashtable is NULL (which can be done by DISCARD + * VARIABLES), we are sure that there aren't any active session variable + * in this session. + */ + if (!sessionvars) + { + list_free(xact_recheck_varids); + xact_recheck_varids = NIL; + return; + } + + elog(DEBUG1, "effective call of sync_sessionvars_all()"); + + /* + * The recheck list can contain many duplicates, so clean it up before + * processing to avoid extraneous work. + */ + list_sort(xact_recheck_varids, list_oid_cmp); + list_deduplicate_oid(xact_recheck_varids); + + /* + * When we check the variables, the system cache can be invalidated, + * and then xact_recheck_varids can be extended. But we need to iterate + * over stable list, and we must not at same time discard invalidation + * messages. + * + * Steps of possible solution: + * + * 1. move xact_recheck_varids to aux variable, and reset + * xact_recheck_varids, + * + * 2. process fields in list of aux variable, + * + * 3. merge the content of aux variable back to xact_recheck_varids. + */ + xact_recheck_varids_snapshot = xact_recheck_varids; + xact_recheck_varids = NIL; + + /* + * This routine is called before any reading, so the session should be in + * transaction state. This is required to access the system catalog. + */ + Assert(IsTransactionState()); + + foreach(l, xact_recheck_varids_snapshot) + { + bool found; + Oid varid = lfirst_oid(l); + + svar = (SVariable) hash_search(sessionvars, &varid, + HASH_FIND, &found); + + /* + * Remove invalid variables, but don't touch variables that were + * dropped by the current top level local transaction or any + * subtransaction underneath, as there's no guarantee that the + * transaction will be committed. Such variables will be removed in + * the next transaction if needed. + */ + if (found) + { + /* + * If this is a variable dropped by the current transaction, + * ignore it and keep the oid to recheck in the next transaction. + */ + if (filter_lxid && svar->drop_lxid == MyProc->lxid) + continue; + + if (!is_session_variable_valid(svar)) + remove_session_variable(svar); + } + + /* + * If caller asked to filter the list, we have to clean items as they + * are processed. + */ + if (filter_lxid) + xact_recheck_varids_snapshot = + foreach_delete_current(xact_recheck_varids_snapshot, + l); + } + + /* + * If caller ask to filter the list, some items are not processed + * and we should to merge these items to xact_recheck_varids. + */ + if (filter_lxid) + { + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(SVariableMemoryContext); + + xact_recheck_varids = list_concat(xact_recheck_varids, + xact_recheck_varids_snapshot); + + MemoryContextSwitchTo(oldcxt); + } + + list_free(xact_recheck_varids_snapshot); +} + +/* + * Create the hash table for storing session variables. + */ +static void +create_sessionvars_hashtables(void) +{ + HASHCTL vars_ctl; + + Assert(!sessionvars); + + /* set callbacks */ + if (!SVariableMemoryContext) + { + /* Read sinval messages */ + CacheRegisterSyscacheCallback(VARIABLEOID, + pg_variable_cache_callback, + (Datum) 0); + + /* We need our own long lived memory context */ + SVariableMemoryContext = + AllocSetContextCreate(TopMemoryContext, + "session variables", + ALLOCSET_START_SMALL_SIZES); + } + + Assert(SVariableMemoryContext); + + memset(&vars_ctl, 0, sizeof(vars_ctl)); + vars_ctl.keysize = sizeof(Oid); + vars_ctl.entrysize = sizeof(SVariableData); + vars_ctl.hcxt = SVariableMemoryContext; + + Assert(sessionvars == NULL); + + sessionvars = hash_create("Session variables", 64, &vars_ctl, + HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); +} + +/* + * Assign some content to the session variable. It's copied to + * SVariableMemoryContext if necessary. + * + * init_mode is true when the value of session variable should be initialized + * by the default expression if any. This is the only case where we allow the + * modification of an immutable variables with default expression. + * + * If any error happens, the existing value shouldn't be modified. + */ +static void +set_session_variable(SVariable svar, Datum value, bool isnull, bool init_mode) +{ + Datum newval = value; + + Assert(svar && OidIsValid(svar->typid)); + + /* Don't allow assignment of null to NOT NULL variable */ + if (isnull && svar->is_not_null) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("null value is not allowed for NOT NULL session variable \"%s.%s\"", + get_namespace_name(get_session_variable_namespace(svar->varid)), + get_session_variable_name(svar->varid)))); + + /* + * Don't allow the modification of an immutable session variable that + * already has an assigned value (possibly NULL) or has a default + * expression (in which case the value should always be the result of + * default expression evaluation) unless the variable is being initialized. + */ + if (!init_mode && + (svar->is_immutable && + (svar->is_valid || svar->has_defexpr))) + ereport(ERROR, + (errcode(ERRCODE_ERROR_IN_ASSIGNMENT), + errmsg("session variable \"%s.%s\" is declared IMMUTABLE", + get_namespace_name(get_session_variable_namespace(svar->varid)), + get_session_variable_name(svar->varid)))); + + if (!isnull) + { + MemoryContext oldcxt = MemoryContextSwitchTo(SVariableMemoryContext); + + newval = datumCopy(value, svar->typbyval, svar->typlen); + + MemoryContextSwitchTo(oldcxt); + } + else + { + /* The caller shouldn't have provided any real value. */ + Assert(value == (Datum) 0); + } + + free_session_variable_value(svar); + + svar->value = newval; + + svar->isnull = isnull; + svar->freeval = newval != value; + svar->is_valid = true; + + /* + * XXX While unlikely, an error here is possible. + * It wouldn't leak memory as the allocated chunk has already been + * correctly assigned to the session variable, but would contradict this + * function contract, which is that this function should either succeed or + * leave the current value untouched. + */ + elog(DEBUG1, "session variable \"%s.%s\" (oid:%u) has new value", + get_namespace_name(get_session_variable_namespace(svar->varid)), + get_session_variable_name(svar->varid), + svar->varid); +} + +/* + * Initialize session variable svar from variable var + */ +static void +init_session_variable(SVariable svar, Variable *var) +{ + MemoryContext oldcxt; + + Assert(OidIsValid(var->oid)); + + svar->varid = var->oid; + svar->create_lsn = var->create_lsn; + + svar->isnull = true; + svar->freeval = false; + svar->value = (Datum) 0; + + svar->typid = var->typid; + get_typlenbyval(var->typid, &svar->typlen, &svar->typbyval); + + svar->is_domain = (get_typtype(var->typid) == TYPTYPE_DOMAIN); + svar->domain_check_extra = NULL; + svar->domain_check_extra_lxid = InvalidLocalTransactionId; + + svar->drop_lxid = InvalidLocalTransactionId; + + svar->is_not_null = var->is_not_null; + svar->is_immutable = var->is_immutable; + svar->has_defexpr = var->has_defexpr; + + /* the value of variable is not known yet */ + svar->is_valid = false; + + svar->hashvalue = GetSysCacheHashValue1(VARIABLEOID, + ObjectIdGetDatum(var->oid)); + + svar->eox_reset = var->eoxaction == VARIABLE_EOX_RESET || + var->eoxaction == VARIABLE_EOX_DROP; + + oldcxt = MemoryContextSwitchTo(TopTransactionContext); + + if (svar->eox_reset) + xact_reset_varids = lappend_oid(xact_reset_varids, var->oid); + + MemoryContextSwitchTo(oldcxt); +} + +/* + * Search a seesion variable in the hash table given its oid. If it + * doesn't exist, then insert it (and calculate defexpr if it exists). + * + * Caller is responsible for doing permission checks. + * + * As side effect this function acquires AccessShareLock on + * related session variable until the end of the transaction. + */ +static SVariable +prepare_variable_for_reading(Oid varid) +{ + SVariable svar; + Variable var; + bool found; + + var.oid = InvalidOid; + + if (!sessionvars) + create_sessionvars_hashtables(); + + /* Protect used session variable against drop until transaction end */ + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); + + /* + * Make sure that all entries in sessionvars hash table are valid, but + * keeping variables dropped by the current transaction. + */ + sync_sessionvars_all(true); + + svar = (SVariable) hash_search(sessionvars, &varid, + HASH_ENTER, &found); + + /* Return content if it is available and valid */ + if (!found || !svar->is_valid) + { + /* We need to load defexpr. */ + InitVariable(&var, varid, false); + + if (!found) + { + init_session_variable(svar, &var); + + elog(DEBUG1, "session variable \"%s.%s\" (oid:%u) has new entry in memory (emitted by READ)", + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid), + varid); + } + + /* + * Raise an error if this is a NOT NULL variable without default + * expression. + */ + if (var.is_not_null && !var.defexpr) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("null value is not allowed for NOT NULL session variable \"%s.%s\"", + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)), + errdetail("The session variable was not initialized yet."))); + + if (svar->has_defexpr) + { + Datum value = (Datum) 0; + bool isnull; + EState *estate = NULL; + Expr *defexpr; + ExprState *defexprs; + MemoryContext oldcxt; + + /* Prepare default expr */ + estate = CreateExecutorState(); + + oldcxt = MemoryContextSwitchTo(estate->es_query_cxt); + + defexpr = expression_planner((Expr *) var.defexpr); + defexprs = ExecInitExpr(defexpr, NULL); + value = ExecEvalExprSwitchContext(defexprs, + GetPerTupleExprContext(estate), + &isnull); + + MemoryContextSwitchTo(oldcxt); + + /* Store result before releasing Executor memory */ + set_session_variable(svar, value, isnull, true); + + FreeExecutorState(estate); + } + else + set_session_variable(svar, (Datum) 0, true, true); + } + + /* + * Although the value of domain type should be valid (it is checked when + * it is assigned to session variable), we have to check related + * constraints each time we access the variable. It can be more expensive + * than in PL/pgSQL, as PL/pgSQL forces domain checks only when the value is assigned + * to the variable or when the value is returned from function. + * However, domain types have a constraint cache so it's not too much + * expensive.. + */ + if (svar->is_domain) + { + /* + * Store domain_check extra in TopTransactionContext. When we are in + * other transaction, the domain_check_extra cache is not valid + * anymore. + */ + if (svar->domain_check_extra_lxid != MyProc->lxid) + svar->domain_check_extra = NULL; + + domain_check(svar->value, svar->isnull, + svar->typid, &svar->domain_check_extra, + TopTransactionContext); + + svar->domain_check_extra_lxid = MyProc->lxid; + } + + return svar; +} + +/* + * Store the given value in an SVariable, and cache it if not already present. + * + * Caller is responsible for doing permission checks. + * + * As side effect this function acquires AccessShareLock on + * related session variable until the end of the transaction. + */ +void +SetSessionVariable(Oid varid, Datum value, bool isNull) +{ + SVariable svar; + bool found; + + if (!sessionvars) + create_sessionvars_hashtables(); + + /* Protect used session variable against drop until transaction end */ + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); + + /* + * Make sure that all entries in sessionvars hash table are valid, but + * keeping variables dropped by the current transaction. + */ + sync_sessionvars_all(true); + + svar = (SVariable) hash_search(sessionvars, &varid, + HASH_ENTER, &found); + + if (!found) + { + Variable var; + + /* We don't need to know defexpr here */ + InitVariable(&var, varid, true); + init_session_variable(svar, &var); + + elog(DEBUG1, "session variable \"%s.%s\" (oid:%u) has new entry in memory (emitted by WRITE)", + get_namespace_name(get_session_variable_namespace(svar->varid)), + get_session_variable_name(svar->varid), + varid); + } + + /* + * This should either succeed or fail without changing the currently stored + * value. + */ + set_session_variable(svar, value, isNull, false); +} + +/* + * Wrapper around SetSessionVariable after checking for correct permission. + */ +void +SetSessionVariableWithSecurityCheck(Oid varid, Datum value, bool isNull) +{ + AclResult aclresult; + + /* + * Is caller allowed to update the session variable? + */ + aclresult = object_aclcheck(VariableRelationId, varid, GetUserId(), ACL_UPDATE); + if (aclresult != ACLCHECK_OK) + aclcheck_error(aclresult, OBJECT_VARIABLE, get_session_variable_name(varid)); + + SetSessionVariable(varid, value, isNull); +} + +/* + * Returns a copy of the value of the session variable specified by its oid. + * Caller is responsible for doing permission checks. + */ +Datum +CopySessionVariable(Oid varid, bool *isNull, Oid *typid) +{ + SVariable svar; + Datum result; + + svar = prepare_variable_for_reading(varid); + Assert(svar != NULL && svar->is_valid); + + *typid = svar->typid; + + /* force copy of non NULL value */ + if (!svar->isnull) + { + result = datumCopy(svar->value, svar->typbyval, svar->typlen); + *isNull = false; + } + else + { + result = (Datum) 0; + *isNull = true; + } + + return (Datum) result; +} + +/* + * Returns a copy of ths value of the session variable specified by its oid + * with a check of the expected type. Like previous CopySessionVariable, the + * caller is responsible for doing permission checks. + */ +Datum +CopySessionVariableWithTypeCheck(Oid varid, bool *isNull, Oid expected_typid) +{ + SVariable svar; + Datum result; + + svar = prepare_variable_for_reading(varid); + Assert(svar != NULL && svar->is_valid); + + if (expected_typid != svar->typid) + elog(ERROR, "type of variable \"%s.%s\" is different than expected", + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)); + + if (!svar->isnull) + { + result = datumCopy(svar->value, svar->typbyval, svar->typlen); + *isNull = false; + } + else + { + result = (Datum) 0; + *isNull = true; + } + + return (Datum) result; +} + +/* + * Do the necessary work to setup local memory management of a new + * variable. + * + * Caller should already have created the necessary entry in catalog + * and made them visible. + */ +void +SessionVariableCreatePostprocess(Oid varid, char eoxaction) +{ + /* + * For temporary variables, we need to create a new end of xact action to + * ensure deletion from catalog. + */ + if (eoxaction == VARIABLE_EOX_DROP) + { + Assert(isTempNamespace(get_session_variable_namespace(varid))); + + register_session_variable_xact_action(varid, SVAR_ON_COMMIT_DROP); + } +} + +/* + * Handle the local memory cleanup for a DROP VARIABLE command. + * + * Caller should take care of removing the pg_variable entry first. + */ +void +SessionVariableDropPostprocess(Oid varid) +{ + /* + * The entry was removed from catalog already, we must not do it + * again at end of xact time. + */ + unregister_session_variable_xact_action(varid, SVAR_ON_COMMIT_DROP); + + if (sessionvars) + { + bool found; + SVariable svar = (SVariable) hash_search(sessionvars, &varid, + HASH_FIND, &found); + + if (found) + { + /* + * Save the current top level local transaction id to make sure we + * don't automatically remove the local variable storage in + * sync_sessionvars_all, as the DROP VARIABLE will send an + * invalidation message. + */ + Assert(LocalTransactionIdIsValid(MyProc->lxid)); + svar->drop_lxid = MyProc->lxid; + + /* + * For variables that are not ON TRANSACTION END RESET, we need to + * register an SVAR_ON_COMMIT_RESET action to free the local + * memory for this variable when the top level transaction + * is committed (we don't need to wait for sinval + * message). The cleanup action for one session variable can be + * duplicated in the action list without causing any problem, so we + * don't need to ensure uniqueness. We need a different action + * from RESET, because RESET is executed on any transaction end, + * but we want to execute this cleanup only when the current + * transaction will be committed. This action can be reverted by + * ABORT of DROP VARIABLE command. + */ + if (!svar->eox_reset) + register_session_variable_xact_action(varid, + SVAR_ON_COMMIT_RESET); + } + } +} + +/* + * Fast drop of the complete content of all session variables hash table, and + * cleanup of any list that wouldn't be relevant anymore. + * This is used by DISCARD VARIABLES (and DISCARD ALL) command. + */ +void +ResetSessionVariables(void) +{ + ListCell *lc; + + /* Destroy hash table and reset related memory context */ + if (sessionvars) + { + hash_destroy(sessionvars); + sessionvars = NULL; + } + + /* Release memory allocated by session variables */ + if (SVariableMemoryContext != NULL) + MemoryContextReset(SVariableMemoryContext); + + /* + * There isn't any session variable left, but we still need to retain the + * ON COMMIT DROP actions if any. + */ + foreach(lc, xact_on_commit_actions) + { + SVariableXActActionItem *xact_ai = + (SVariableXActActionItem *) lfirst(lc); + + if (xact_ai->action == SVAR_ON_COMMIT_DROP) + continue; + + pfree(xact_ai); + xact_on_commit_actions = foreach_delete_current(xact_on_commit_actions, + lc); + } + + /* We should clean xact_reset_varids */ + list_free(xact_reset_varids); + xact_reset_varids = NIL; + + /* + * xact_recheck_varids is stored in SVariableMemoryContext, so it has + * already been freed, just reset the list. + */ + xact_recheck_varids = NIL; +} + +/* + * Perform the necessary work for ON TRANSACTION END RESET and ON COMMIT DROP + * session variables. + * If the transaction is committed, also process the delayed memory cleanup of + * local DROP VARIABLE and process all pending rechecks. */ void AtPreEOXact_SessionVariable(bool isCommit) { ListCell *l; - foreach(l, xact_drop_actions) + /* + * Clean memory for all ON TRANSACTION END RESET variables. Do it first, + * as it reduces the overhead of the RECHECK action list. + */ + foreach(l, xact_reset_varids) { - SVariableXActActionItem *xact_ai = - (SVariableXActActionItem *) lfirst(l); + remove_session_variable_by_id(lfirst_oid(l)); + } - /* Iterate only over entries that are still pending */ - if (xact_ai->deleting_subid == InvalidSubTransactionId) + /* We can now clean xact_reset_varids */ + list_free(xact_reset_varids); + xact_reset_varids = NIL; + + if (isCommit && xact_on_commit_actions) + { + foreach(l, xact_on_commit_actions) { + SVariableXActActionItem *xact_ai = + (SVariableXActActionItem *) lfirst(l); + + /* Iterate only over entries that are still pending */ + if (xact_ai->deleting_subid != InvalidSubTransactionId) + continue; /* - * ON COMMIT DROP is allowed only for temp session variables. So - * we should explicitly delete only when current transaction was - * committed. When it's rollback, then session variable is removed - * automatically. + * ON COMMIT DROP is allowed only for temp session variables. + * So we should explicitly delete only when the current + * transaction is committed. When it's rollbacked, the session + * variable is removed automatically. */ - if (isCommit) + if (xact_ai->action == SVAR_ON_COMMIT_DROP) { ObjectAddress object; @@ -191,10 +1150,30 @@ AtPreEOXact_SessionVariable(bool isCommit) elog(DEBUG1, "session variable (oid:%u) will be deleted (forced by SVAR_ON_COMMIT_DROP action)", xact_ai->varid); + /* + * If the variable was locally set, the memory will be + * automatically cleaned up when we process the underlying + * shared invalidation for this drop. There can't be a recheck + * action for this variable, so there's nothing to gain + * explicitly removing it here. + */ performDeletion(&object, DROP_CASCADE, PERFORM_DELETION_INTERNAL | PERFORM_DELETION_QUIETLY); } + else + { + /* + * When we process DROP VARIABLE statement issued by the + * current transaction, we create an SVAR_ON_COMMIT_RESET xact + * action. We want to process this action only when related + * transaction is commited (when DROP VARIABLE statement + * sucessfully processed) as we need to preserve the variable + * content if the transaction that issued the DROP VARAIBLE + * statement is rollbacked. + */ + remove_session_variable_by_id(xact_ai->varid); + } } } @@ -202,12 +1181,26 @@ AtPreEOXact_SessionVariable(bool isCommit) * Any drop action left is an entry that was unregistered and not * rollbacked, so we can simply remove them. */ - list_free_deep(xact_drop_actions); - xact_drop_actions = NIL; + list_free_deep(xact_on_commit_actions); + xact_on_commit_actions = NIL; + + /* + * We process the list of recheck last for performance reason,the previous + * steps might remove entries from the hash table. + * We need catalog access to process the recheck, so this can only be done + * if the transaction is committed. Otherwise, we just keep the recheck + * list as-is and it will be processed at the next (committed) transaction. + */ + if (isCommit && xact_recheck_varids) + { + Assert(sessionvars); + + sync_sessionvars_all(false); + } } /* - * Post-subcommit or post-subabort cleanup of xact action list. + * Post-subcommit or post-subabort cleanup of xact_on_commit_actions list. * * During subabort, we can immediately remove entries created during this * subtransaction. During subcommit, just transfer entries marked during @@ -220,24 +1213,130 @@ AtEOSubXact_SessionVariable(bool isCommit, { ListCell *cur_item; - foreach(cur_item, xact_drop_actions) + foreach(cur_item, xact_on_commit_actions) { SVariableXActActionItem *xact_ai = (SVariableXActActionItem *) lfirst(cur_item); + /* + * The subtransaction that created this entry was rollbacked, we can + * remove it. + */ if (!isCommit && xact_ai->creating_subid == mySubid) { /* cur_item must be removed */ - xact_drop_actions = foreach_delete_current(xact_drop_actions, cur_item); + xact_on_commit_actions = foreach_delete_current(xact_on_commit_actions, cur_item); pfree(xact_ai); } else { - /* cur_item must be preserved */ + /* Otherwise cur_item must be preserved */ if (xact_ai->creating_subid == mySubid) xact_ai->creating_subid = parentSubid; if (xact_ai->deleting_subid == mySubid) - xact_ai->deleting_subid = isCommit ? parentSubid : InvalidSubTransactionId; + xact_ai->deleting_subid = isCommit ? parentSubid + : InvalidSubTransactionId; + } + } +} + +/* + * pg_session_variables - designed for testing + * + * This is a function designed for testing and debugging. It returns the + * content of sessionvars as-is, and can therefore display entries about + * session variables that were dropped but for which this backend didn't + * process the shared invalidations yet. + */ +Datum +pg_session_variables(PG_FUNCTION_ARGS) +{ +#define NUM_PG_SESSION_VARIABLES_ATTS 10 + + elog(DEBUG1, "pg_session_variables start"); + + InitMaterializedSRF(fcinfo, 0); + + if (sessionvars) + { + ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; + HASH_SEQ_STATUS status; + SVariable svar; + + /* + * Make sure that all entries in sessionvars hash table are valid, but + * keeping variables dropped by the current transaction. + */ + sync_sessionvars_all(true); + + hash_seq_init(&status, sessionvars); + + while ((svar = (SVariable) hash_seq_search(&status)) != NULL) + { + Datum values[NUM_PG_SESSION_VARIABLES_ATTS]; + bool nulls[NUM_PG_SESSION_VARIABLES_ATTS]; + HeapTuple tp; + + memset(values, 0, sizeof(values)); + memset(nulls, 0, sizeof(nulls)); + + values[0] = ObjectIdGetDatum(svar->varid); + values[3] = ObjectIdGetDatum(svar->typid); + + /* check if session variable is visible in system catalog */ + tp = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(svar->varid)); + + /* + * Sessionvars can hold data of variables removed from catalog, + * (and not purged) and then namespacename and name cannot be read + * from catalog. + */ + if (HeapTupleIsValid(tp)) + { + Form_pg_variable varform = (Form_pg_variable) GETSTRUCT(tp); + + /* When we see data in catalog */ + values[1] = CStringGetTextDatum( + get_namespace_name(varform->varnamespace)); + + values[2] = CStringGetTextDatum(NameStr(varform->varname)); + + values[4] = CStringGetTextDatum(format_type_be(svar->typid)); + values[5] = BoolGetDatum(false); + values[6] = BoolGetDatum(svar->is_valid); + + values[8] = BoolGetDatum( + object_aclcheck(VariableRelationId, svar->varid, + GetUserId(), ACL_SELECT) == ACLCHECK_OK); + + values[9] = BoolGetDatum( + object_aclcheck(VariableRelationId, svar->varid, + GetUserId(), ACL_UPDATE) == ACLCHECK_OK); + + ReleaseSysCache(tp); + } + else + { + /* + * When session variable was removed from catalog, but we + * haven't processed the invlidation yet. + */ + nulls[1] = true; + values[2] = CStringGetTextDatum( + DatumGetCString(DirectFunctionCall1(oidout, svar->varid))); + values[4] = PointerGetDatum( + cstring_to_text(format_type_be(svar->typid))); + values[5] = BoolGetDatum(true); + values[6] = BoolGetDatum(svar->is_valid); + nulls[7] = true; + nulls[8] = true; + } + + tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls); } } + + elog(DEBUG1, "pg_session_variables end"); + + return (Datum) 0; } diff --git a/src/backend/executor/execExpr.c b/src/backend/executor/execExpr.c index bf257a41c8..4881c0c90b 100644 --- a/src/backend/executor/execExpr.c +++ b/src/backend/executor/execExpr.c @@ -34,6 +34,7 @@ #include "catalog/objectaccess.h" #include "catalog/pg_proc.h" #include "catalog/pg_type.h" +#include "commands/session_variable.h" #include "executor/execExpr.h" #include "executor/nodeSubplan.h" #include "funcapi.h" @@ -983,6 +984,81 @@ ExecInitExprRec(Expr *node, ExprState *state, scratch.d.param.paramtype = param->paramtype; ExprEvalPushStep(state, &scratch); break; + + case PARAM_VARIABLE: + { + int es_num_session_variables = 0; + SessionVariableValue *es_session_variables = NULL; + + if (state->parent && state->parent->state) + { + es_session_variables = state->parent->state->es_session_variables; + es_num_session_variables = state->parent->state->es_num_session_variables; + } + + if (es_session_variables) + { + SessionVariableValue *var; + + /* + * Use buffered session variables when the + * buffer with copied values is avaiable + * (standard query executor mode) + */ + + /* Parameter sanity checks. */ + if (param->paramid >= es_num_session_variables) + elog(ERROR, "paramid of PARAM_VARIABLE param is out of range"); + + var = &es_session_variables[param->paramid]; + + if (var->typid != param->paramtype) + elog(ERROR, "type of buffered value is different than PARAM_VARIABLE type"); + + /* + * In this case, pass the value like + * a constant. + */ + scratch.opcode = EEOP_CONST; + scratch.d.constval.value = var->value; + scratch.d.constval.isnull = var->isnull; + ExprEvalPushStep(state, &scratch); + } + else + { + AclResult aclresult; + Oid varid = param->paramvarid; + Oid vartype = param->paramtype; + + /* + * When the expression is evaluated directly + * without query executor start (plpgsql simple + * expr evaluation), then the array es_session_variables + * is null. In this case we need to use direct + * access to session variables. The values are + * not protected by using copy, but it is not + * problem (we don't need to emulate stability + * of the value). + * + * In this case we should to do aclcheck, because + * usual aclcheck from standard_ExecutorStart + * is not executed in this case. Fortunately + * it is just once per transaction. + */ + aclresult = object_aclcheck(VariableRelationId, varid, + GetUserId(), ACL_SELECT); + if (aclresult != ACLCHECK_OK) + aclcheck_error(aclresult, OBJECT_VARIABLE, + get_session_variable_name(varid)); + + scratch.opcode = EEOP_PARAM_VARIABLE; + scratch.d.vparam.varid = varid; + scratch.d.vparam.vartype = vartype; + ExprEvalPushStep(state, &scratch); + } + } + break; + case PARAM_EXTERN: /* diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c index 7cc443ec52..c7d309a7f5 100644 --- a/src/backend/executor/execExprInterp.c +++ b/src/backend/executor/execExprInterp.c @@ -59,6 +59,7 @@ #include "access/heaptoast.h" #include "catalog/pg_type.h" #include "commands/sequence.h" +#include "commands/session_variable.h" #include "executor/execExpr.h" #include "executor/nodeSubplan.h" #include "funcapi.h" @@ -449,6 +450,7 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull) &&CASE_EEOP_PARAM_EXEC, &&CASE_EEOP_PARAM_EXTERN, &&CASE_EEOP_PARAM_CALLBACK, + &&CASE_EEOP_PARAM_VARIABLE, &&CASE_EEOP_CASE_TESTVAL, &&CASE_EEOP_MAKE_READONLY, &&CASE_EEOP_IOCOERCE, @@ -1087,6 +1089,21 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull) EEO_NEXT(); } + EEO_CASE(EEOP_PARAM_VARIABLE) + { + /* + * direct access to session variable (without buffering). + * Because returned value can be used (without an assignement) + * after the referenced session variables is updated, we have + * to return copy of stored value every time. This is not an + * issue for local (plpgsql) variables. + */ + *op->resvalue = CopySessionVariableWithTypeCheck(op->d.vparam.varid, + op->resnull, + op->d.vparam.vartype); + EEO_NEXT(); + } + EEO_CASE(EEOP_CASE_TESTVAL) { /* diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index 0186be452c..a407b53038 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -48,6 +48,7 @@ #include "catalog/pg_publication.h" #include "commands/matview.h" #include "commands/trigger.h" +#include "commands/session_variable.h" #include "executor/execdebug.h" #include "executor/nodeSubplan.h" #include "foreign/fdwapi.h" @@ -201,6 +202,63 @@ standard_ExecutorStart(QueryDesc *queryDesc, int eflags) Assert(queryDesc->sourceText != NULL); estate->es_sourceText = queryDesc->sourceText; + /* + * The executor doesn't work with session variables directly. Values of + * related session variables are copied to dedicated array, and this array + * is passed to executor. + */ + if (queryDesc->num_session_variables > 0) + { + /* + * When a parallel query needs to access query parameters (including + * related session variables), then related session variables are + * restored (deserialized) in queryDesc already. So just push pointer + * of this array to executor's estate. + */ + Assert(IsParallelWorker()); + estate->es_session_variables = queryDesc->session_variables; + estate->es_num_session_variables = queryDesc->num_session_variables; + } + else if (queryDesc->plannedstmt->sessionVariables) + { + ListCell *lc; + int nSessionVariables; + int i = 0; + + /* + * In this case, the query uses session variables, but we have to + * prepare the array with passed values (of used session variables) + * first. + */ + Assert(!IsParallelWorker()); + nSessionVariables = list_length(queryDesc->plannedstmt->sessionVariables); + + /* Create the array used for passing values of used session variables */ + estate->es_session_variables = (SessionVariableValue *) + palloc(nSessionVariables * sizeof(SessionVariableValue)); + + /* Fill the array */ + foreach(lc, queryDesc->plannedstmt->sessionVariables) + { + AclResult aclresult; + Oid varid = lfirst_oid(lc); + + aclresult = object_aclcheck(VariableRelationId, varid, GetUserId(), ACL_SELECT); + if (aclresult != ACLCHECK_OK) + aclcheck_error(aclresult, OBJECT_VARIABLE, + get_session_variable_name(varid)); + + estate->es_session_variables[i].varid = varid; + estate->es_session_variables[i].value = CopySessionVariable(varid, + &estate->es_session_variables[i].isnull, + &estate->es_session_variables[i].typid); + + i++; + } + + estate->es_num_session_variables = nSessionVariables; + } + /* * Fill in the query environment, if any, from queryDesc. */ diff --git a/src/backend/executor/execParallel.c b/src/backend/executor/execParallel.c index cc2b8ccab7..64e3b66e73 100644 --- a/src/backend/executor/execParallel.c +++ b/src/backend/executor/execParallel.c @@ -12,8 +12,9 @@ * workers and ensuring that their state generally matches that of the * leader; see src/backend/access/transam/README.parallel for details. * However, we must save and restore relevant executor state, such as - * any ParamListInfo associated with the query, buffer/WAL usage info, and - * the actual plan to be passed down to the worker. + * any ParamListInfo associated with the query, buffer/WAL usage info, + * session variables buffer, and the actual plan to be passed down to + * the worker. * * IDENTIFICATION * src/backend/executor/execParallel.c @@ -66,6 +67,7 @@ #define PARALLEL_KEY_QUERY_TEXT UINT64CONST(0xE000000000000008) #define PARALLEL_KEY_JIT_INSTRUMENTATION UINT64CONST(0xE000000000000009) #define PARALLEL_KEY_WAL_USAGE UINT64CONST(0xE00000000000000A) +#define PARALLEL_KEY_SESSION_VARIABLES UINT64CONST(0xE00000000000000B) #define PARALLEL_TUPLE_QUEUE_SIZE 65536 @@ -140,6 +142,12 @@ static bool ExecParallelRetrieveInstrumentation(PlanState *planstate, /* Helper function that runs in the parallel worker. */ static DestReceiver *ExecParallelGetReceiver(dsm_segment *seg, shm_toc *toc); +/* Helper functions that can pass values of session variables */ +static Size EstimateSessionVariables(EState *estate); +static void SerializeSessionVariables(EState *estate, char **start_address); +static SessionVariableValue *RestoreSessionVariables(char **start_address, + int *num_session_variables); + /* * Create a serialized representation of the plan to be sent to each worker. */ @@ -598,6 +606,7 @@ ExecInitParallelPlan(PlanState *planstate, EState *estate, char *pstmt_data; char *pstmt_space; char *paramlistinfo_space; + char *session_variables_space; BufferUsage *bufusage_space; WalUsage *walusage_space; SharedExecutorInstrumentation *instrumentation = NULL; @@ -607,6 +616,7 @@ ExecInitParallelPlan(PlanState *planstate, EState *estate, int instrumentation_len = 0; int jit_instrumentation_len = 0; int instrument_offset = 0; + int session_variables_len = 0; Size dsa_minsize = dsa_minimum_size(); char *query_string; int query_len; @@ -662,6 +672,11 @@ ExecInitParallelPlan(PlanState *planstate, EState *estate, shm_toc_estimate_chunk(&pcxt->estimator, paramlistinfo_len); shm_toc_estimate_keys(&pcxt->estimator, 1); + /* Estimate space for serialized session variables. */ + session_variables_len = EstimateSessionVariables(estate); + shm_toc_estimate_chunk(&pcxt->estimator, session_variables_len); + shm_toc_estimate_keys(&pcxt->estimator, 1); + /* * Estimate space for BufferUsage. * @@ -756,6 +771,11 @@ ExecInitParallelPlan(PlanState *planstate, EState *estate, shm_toc_insert(pcxt->toc, PARALLEL_KEY_PARAMLISTINFO, paramlistinfo_space); SerializeParamList(estate->es_param_list_info, ¶mlistinfo_space); + /* Store serialized session variables. */ + session_variables_space = shm_toc_allocate(pcxt->toc, session_variables_len); + shm_toc_insert(pcxt->toc, PARALLEL_KEY_SESSION_VARIABLES, session_variables_space); + SerializeSessionVariables(estate, &session_variables_space); + /* Allocate space for each worker's BufferUsage; no need to initialize. */ bufusage_space = shm_toc_allocate(pcxt->toc, mul_size(sizeof(BufferUsage), pcxt->nworkers)); @@ -1403,6 +1423,7 @@ ParallelQueryMain(dsm_segment *seg, shm_toc *toc) SharedJitInstrumentation *jit_instrumentation; int instrument_options = 0; void *area_space; + char *sessionvariable_space; dsa_area *area; ParallelWorkerContext pwcxt; @@ -1428,6 +1449,14 @@ ParallelQueryMain(dsm_segment *seg, shm_toc *toc) area_space = shm_toc_lookup(toc, PARALLEL_KEY_DSA, false); area = dsa_attach_in_place(area_space, seg); + /* Reconstruct session variables. */ + sessionvariable_space = shm_toc_lookup(toc, + PARALLEL_KEY_SESSION_VARIABLES, + false); + queryDesc->session_variables = + RestoreSessionVariables(&sessionvariable_space, + &queryDesc->num_session_variables); + /* Start up the executor */ queryDesc->plannedstmt->jitFlags = fpes->jit_flags; ExecutorStart(queryDesc, fpes->eflags); @@ -1496,3 +1525,117 @@ ParallelQueryMain(dsm_segment *seg, shm_toc *toc) FreeQueryDesc(queryDesc); receiver->rDestroy(receiver); } + +/* + * Estimate the amount of space required to serialize a session variable. + */ +static Size +EstimateSessionVariables(EState *estate) +{ + int i; + Size sz = sizeof(int); + + if (estate->es_session_variables == NULL) + return sz; + + for (i = 0; i < estate->es_num_session_variables; i++) + { + SessionVariableValue *svarval; + Oid typeOid; + int16 typLen; + bool typByVal; + + svarval = &estate->es_session_variables[i]; + + typeOid = svarval->typid; + + sz = add_size(sz, sizeof(Oid)); /* space for type OID */ + + /* space for datum/isnull */ + Assert(OidIsValid(typeOid)); + get_typlenbyval(typeOid, &typLen, &typByVal); + + sz = add_size(sz, + datumEstimateSpace(svarval->value, svarval->isnull, typByVal, typLen)); + } + + return sz; +} + +/* + * Serialize a session variables buffer into caller-provided storage. + * + * We write the number of parameters first, as a 4-byte integer, and then + * write details for each parameter in turn. The details for each parameter + * consist of a 4-byte type OID, and then the datum as serialized by + * datumSerialize(). The caller is responsible for ensuring that there is + * enough storage to store the number of bytes that will be written; use + * EstimateSessionVariables to find out how many will be needed. + * *start_address is updated to point to the byte immediately following those + * written. + * + * RestoreSessionVariables can be used to recreate a session variable buffer + * based on the serialized representation; + */ +static void +SerializeSessionVariables(EState *estate, char **start_address) +{ + int nparams; + int i; + + /* Write number of parameters. */ + nparams = estate->es_num_session_variables; + memcpy(*start_address, &nparams, sizeof(int)); + *start_address += sizeof(int); + + /* Write each parameter in turn. */ + for (i = 0; i < nparams; i++) + { + SessionVariableValue *svarval; + Oid typeOid; + int16 typLen; + bool typByVal; + + svarval = &estate->es_session_variables[i]; + typeOid = svarval->typid; + + /* Write type OID. */ + memcpy(*start_address, &typeOid, sizeof(Oid)); + *start_address += sizeof(Oid); + + Assert(OidIsValid(typeOid)); + get_typlenbyval(typeOid, &typLen, &typByVal); + + datumSerialize(svarval->value, svarval->isnull, typByVal, typLen, + start_address); + } +} + +static SessionVariableValue * +RestoreSessionVariables(char **start_address, int *num_session_variables) +{ + SessionVariableValue *session_variables; + int i; + int nparams; + + memcpy(&nparams, *start_address, sizeof(int)); + *start_address += sizeof(int); + + *num_session_variables = nparams; + session_variables = (SessionVariableValue *) + palloc(nparams * sizeof(SessionVariableValue)); + + for (i = 0; i < nparams; i++) + { + SessionVariableValue *svarval = &session_variables[i]; + + /* Read type OID. */ + memcpy(&svarval->typid, *start_address, sizeof(Oid)); + *start_address += sizeof(Oid); + + /* Read datum/isnull. */ + svarval->value = datumRestore(start_address, &svarval->isnull); + } + + return session_variables; +} diff --git a/src/backend/jit/llvm/llvmjit_expr.c b/src/backend/jit/llvm/llvmjit_expr.c index 774db57ae2..4ec8bc8d9d 100644 --- a/src/backend/jit/llvm/llvmjit_expr.c +++ b/src/backend/jit/llvm/llvmjit_expr.c @@ -1073,6 +1073,12 @@ llvm_compile_expr(ExprState *state) LLVMBuildBr(b, opblocks[opno + 1]); break; + case EEOP_PARAM_VARIABLE: + build_EvalXFunc(b, mod, "ExecEvalParamVariable", + v_state, op, v_econtext); + LLVMBuildBr(b, opblocks[opno + 1]); + break; + case EEOP_PARAM_CALLBACK: { LLVMTypeRef v_functype; diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 1e4dd27dba..4157f755c9 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -321,6 +321,7 @@ standard_planner(Query *parse, const char *query_string, int cursorOptions, glob->lastPlanNodeId = 0; glob->transientPlan = false; glob->dependsOnRole = false; + glob->sessionVariables = NIL; /* * Assess whether it's feasible to use parallel mode for this query. We @@ -534,6 +535,7 @@ standard_planner(Query *parse, const char *query_string, int cursorOptions, result->paramExecTypes = glob->paramExecTypes; /* utilityStmt should be null, but we might as well copy it */ result->utilityStmt = parse->utilityStmt; + result->sessionVariables = glob->sessionVariables; result->stmt_location = parse->stmt_location; result->stmt_len = parse->stmt_len; @@ -700,6 +702,12 @@ subquery_planner(PlannerGlobal *glob, Query *parse, */ pull_up_subqueries(root); + /* + * Check if some subquery uses session variable. Flag hasSessionVariables + * should be true if query or some subquery uses any session variable. + */ + pull_up_has_session_variables(root); + /* * If this is a simple UNION ALL query, flatten it into an appendrel. We * do this now because it requires applying pull_up_subqueries to the leaf diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index f6f8a79354..f18ed8ef9a 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -211,6 +211,8 @@ static List *set_returning_clause_references(PlannerInfo *root, static List *set_windowagg_runcondition_references(PlannerInfo *root, List *runcondition, Plan *plan); +static bool pull_up_has_session_variables_walker(Node *node, + PlannerInfo *root); /***************************************************************************** @@ -1294,6 +1296,50 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) return plan; } +/* + * Search usage of session variables in subqueries + */ +void +pull_up_has_session_variables(PlannerInfo *root) +{ + Query *query = root->parse; + + if (query->hasSessionVariables) + { + root->hasSessionVariables = true; + } + else + { + (void) query_tree_walker(query, + pull_up_has_session_variables_walker, + (void *) root, 0); + } +} + +static bool +pull_up_has_session_variables_walker(Node *node, PlannerInfo *root) +{ + if (node == NULL) + return false; + if (IsA(node, Query)) + { + Query *query = (Query *) node; + + if (query->hasSessionVariables) + { + root->hasSessionVariables = true; + return false; + } + + /* Recurse into subselects */ + return query_tree_walker((Query *) node, + pull_up_has_session_variables_walker, + (void *) root, 0); + } + return expression_tree_walker(node, pull_up_has_session_variables_walker, + (void *) root); +} + /* * set_indexonlyscan_references * Do set_plan_references processing on an IndexOnlyScan @@ -1931,8 +1977,9 @@ copyVar(Var *var) * This is code that is common to all variants of expression-fixing. * We must look up operator opcode info for OpExpr and related nodes, * add OIDs from regclass Const nodes into root->glob->relationOids, and - * add PlanInvalItems for user-defined functions into root->glob->invalItems. - * We also fill in column index lists for GROUPING() expressions. + * add PlanInvalItems for user-defined functions and session variables into + * root->glob->invalItems. We also fill in column index lists for GROUPING() + * expressions. * * We assume it's okay to update opcode info in-place. So this could possibly * scribble on the planner's input data structures, but it's OK. @@ -2022,15 +2069,39 @@ fix_expr_common(PlannerInfo *root, Node *node) g->cols = cols; } } + else if (IsA(node, Param)) + { + Param *p = (Param *) node; + + if (p->paramkind == PARAM_VARIABLE) + { + PlanInvalItem *inval_item = makeNode(PlanInvalItem); + + /* paramid is still session variable id */ + inval_item->cacheId = VARIABLEOID; + inval_item->hashValue = GetSysCacheHashValue1(VARIABLEOID, + ObjectIdGetDatum(p->paramvarid)); + + /* Append this variable to global, register dependency */ + root->glob->invalItems = lappend(root->glob->invalItems, + inval_item); + } + } } /* * fix_param_node * Do set_plan_references processing on a Param + * Collect session variables list and replace variable oid by + * index to collected list. * * If it's a PARAM_MULTIEXPR, replace it with the appropriate Param from * root->multiexpr_params; otherwise no change is needed. * Just for paranoia's sake, we make a copy of the node in either case. + * + * If it's a PARAM_VARIABLE, then we collect used session variables in + * list root->glob->sessionVariable. We should to assign Param paramvarid + * too, and it is position of related session variable in mentioned list. */ static Node * fix_param_node(PlannerInfo *root, Param *p) @@ -2049,6 +2120,41 @@ fix_param_node(PlannerInfo *root, Param *p) elog(ERROR, "unexpected PARAM_MULTIEXPR ID: %d", p->paramid); return copyObject(list_nth(params, colno - 1)); } + + if (p->paramkind == PARAM_VARIABLE) + { + ListCell *lc; + int n = 0; + bool found = false; + + /* We will modify object */ + p = (Param *) copyObject(p); + + /* + * Now, we can actualize list of session variables, and we can + * complete paramid parameter. + */ + foreach(lc, root->glob->sessionVariables) + { + if (lfirst_oid(lc) == p->paramvarid) + { + p->paramid = n; + found = true; + break; + } + n += 1; + } + + if (!found) + { + root->glob->sessionVariables = lappend_oid(root->glob->sessionVariables, + p->paramvarid); + p->paramid = n; + } + + return (Node *) p; + } + return (Node *) copyObject(p); } @@ -2110,7 +2216,10 @@ fix_alternative_subplan(PlannerInfo *root, AlternativeSubPlan *asplan, * replacing Aggref nodes that should be replaced by initplan output Params, * choosing the best implementation for AlternativeSubPlans, * looking up operator opcode info for OpExpr and related nodes, - * and adding OIDs from regclass Const nodes into root->glob->relationOids. + * adding OIDs from regclass Const nodes into root->glob->relationOids, + * and assigning paramvarid to PARAM_VARIABLE params, and collecting + * of OIDs of session variables in root->glob->sessionVariables list + * (paramvarid is an position of related session variable in this list). * * 'node': the expression to be modified * 'rtoffset': how much to increment varnos by @@ -2132,7 +2241,8 @@ fix_scan_expr(PlannerInfo *root, Node *node, int rtoffset, double num_exec) root->multiexpr_params != NIL || root->glob->lastPHId != 0 || root->minmax_aggs != NIL || - root->hasAlternativeSubPlans) + root->hasAlternativeSubPlans || + root->hasSessionVariables) { return fix_scan_expr_mutator(node, &context); } diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c index 2f589b1b99..799033da11 100644 --- a/src/backend/optimizer/prep/prepjointree.c +++ b/src/backend/optimizer/prep/prepjointree.c @@ -1265,6 +1265,9 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, /* If subquery had any RLS conditions, now main query does too */ parse->hasRowSecurity |= subquery->hasRowSecurity; + /* If subquery had session variables, now main query does too */ + parse->hasSessionVariables |= subquery->hasSessionVariables; + /* * subquery won't be pulled up if it hasAggs, hasWindowFuncs, or * hasTargetSRFs, so no work needed on those flags diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c index 7f453b04f8..ece03034dd 100644 --- a/src/backend/optimizer/util/clauses.c +++ b/src/backend/optimizer/util/clauses.c @@ -26,6 +26,7 @@ #include "catalog/pg_operator.h" #include "catalog/pg_proc.h" #include "catalog/pg_type.h" +#include "commands/session_variable.h" #include "executor/executor.h" #include "executor/functions.h" #include "funcapi.h" @@ -840,16 +841,17 @@ max_parallel_hazard_walker(Node *node, max_parallel_hazard_context *context) /* * We can't pass Params to workers at the moment either, so they are also - * parallel-restricted, unless they are PARAM_EXTERN Params or are - * PARAM_EXEC Params listed in safe_param_ids, meaning they could be - * either generated within workers or can be computed by the leader and - * then their value can be passed to workers. + * parallel-restricted, unless they are PARAM_EXTERN or PARAM_VARIABLE + * Params or are PARAM_EXEC Params listed in safe_param_ids, meaning they + * could be either generated within workers or can be computed by the + * leader and then their value can be passed to workers. */ else if (IsA(node, Param)) { Param *param = (Param *) node; - if (param->paramkind == PARAM_EXTERN) + if (param->paramkind == PARAM_EXTERN || + param->paramkind == PARAM_VARIABLE) return false; if (param->paramkind != PARAM_EXEC || @@ -2306,6 +2308,7 @@ convert_saop_to_hashed_saop_walker(Node *node, void *context) * value of the Param. * 2. Fold stable, as well as immutable, functions to constants. * 3. Reduce PlaceHolderVar nodes to their contained expressions. + * 4. Current value of session variable can be used for estimation too. *-------------------- */ Node * @@ -2428,6 +2431,29 @@ eval_const_expressions_mutator(Node *node, } } } + else if (param->paramkind == PARAM_VARIABLE && + context->estimate) + { + int16 typLen; + bool typByVal; + Datum pval; + bool isnull; + + get_typlenbyval(param->paramtype, + &typLen, &typByVal); + + pval = CopySessionVariableWithTypeCheck(param->paramvarid, + &isnull, + param->paramtype); + + return (Node *) makeConst(param->paramtype, + param->paramtypmod, + param->paramcollid, + (int) typLen, + pval, + isnull, + typByVal); + } /* * Not replaceable, so just copy the Param (no need to @@ -4828,21 +4854,43 @@ substitute_actual_parameters_mutator(Node *node, { if (node == NULL) return NULL; + + /* + * SQL functions can contain two different kind of params. The nodes with + * paramkind PARAM_EXTERN are related to function's arguments (and should + * be replaced in this step), because this is how we apply the function's + * arguments for an expression. + * + * The nodes with paramkind PARAM_VARIABLE are related to usage of session + * variables. The values of session variables are not passed to expression + * by expression arguments, so it should not be replaced here by + * function's arguments. Although we could substitute params related to + * immutable session variables with default expression by this default + * expression, it is safer to not do it. This way we don't have to run + * security checks here. There can be some performance loss, but an access + * to session variable is fast (and the result of default expression is + * immediately materialized and can be reused). + */ if (IsA(node, Param)) { Param *param = (Param *) node; - if (param->paramkind != PARAM_EXTERN) + if (param->paramkind != PARAM_EXTERN && + param->paramkind != PARAM_VARIABLE) elog(ERROR, "unexpected paramkind: %d", (int) param->paramkind); - if (param->paramid <= 0 || param->paramid > context->nargs) - elog(ERROR, "invalid paramid: %d", param->paramid); - /* Count usage of parameter */ - context->usecounts[param->paramid - 1]++; + if (param->paramkind == PARAM_EXTERN) + { + if (param->paramid <= 0 || param->paramid > context->nargs) + elog(ERROR, "invalid paramid: %d", param->paramid); + + /* Count usage of parameter */ + context->usecounts[param->paramid - 1]++; - /* Select the appropriate actual arg and replace the Param with it */ - /* We don't need to copy at this time (it'll get done later) */ - return list_nth(context->args, param->paramid - 1); + /* Select the appropriate actual arg and replace the Param with it */ + /* We don't need to copy at this time (it'll get done later) */ + return list_nth(context->args, param->paramid - 1); + } } return expression_tree_mutator(node, substitute_actual_parameters_mutator, (void *) context); diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 70932dba61..c2f4f4836f 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -527,6 +527,8 @@ transformDeleteStmt(ParseState *pstate, DeleteStmt *stmt) qry->hasTargetSRFs = pstate->p_hasTargetSRFs; qry->hasAggs = pstate->p_hasAggs; + qry->hasSessionVariables = pstate->p_hasSessionVariables; + assign_query_collations(pstate, qry); /* this must be done after collations, for reliable comparison of exprs */ @@ -951,6 +953,7 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) qry->hasTargetSRFs = pstate->p_hasTargetSRFs; qry->hasSubLinks = pstate->p_hasSubLinks; + qry->hasSessionVariables = pstate->p_hasSessionVariables; assign_query_collations(pstate, qry); @@ -1405,6 +1408,7 @@ transformSelectStmt(ParseState *pstate, SelectStmt *stmt) qry->hasWindowFuncs = pstate->p_hasWindowFuncs; qry->hasTargetSRFs = pstate->p_hasTargetSRFs; qry->hasAggs = pstate->p_hasAggs; + qry->hasSessionVariables = pstate->p_hasSessionVariables; foreach(l, stmt->lockingClause) { @@ -1631,6 +1635,7 @@ transformValuesClause(ParseState *pstate, SelectStmt *stmt) qry->jointree = makeFromExpr(pstate->p_joinlist, NULL); qry->hasSubLinks = pstate->p_hasSubLinks; + qry->hasSessionVariables = pstate->p_hasSessionVariables; assign_query_collations(pstate, qry); @@ -1881,6 +1886,7 @@ transformSetOperationStmt(ParseState *pstate, SelectStmt *stmt) qry->hasWindowFuncs = pstate->p_hasWindowFuncs; qry->hasTargetSRFs = pstate->p_hasTargetSRFs; qry->hasAggs = pstate->p_hasAggs; + qry->hasSessionVariables = pstate->p_hasSessionVariables; foreach(l, lockingClause) { @@ -2421,6 +2427,7 @@ transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt) qry->hasTargetSRFs = pstate->p_hasTargetSRFs; qry->hasSubLinks = pstate->p_hasSubLinks; + qry->hasSessionVariables = pstate->p_hasSessionVariables; assign_query_collations(pstate, qry); diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 327c5956b2..03b03dc142 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -35,16 +35,18 @@ #include "parser/parse_relation.h" #include "parser/parse_target.h" #include "parser/parse_type.h" +#include "storage/lmgr.h" #include "utils/builtins.h" #include "utils/date.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" #include "utils/timestamp.h" +#include "utils/typcache.h" #include "utils/xml.h" /* GUC parameters */ bool Transform_null_equals = false; - +bool session_variables_ambiguity_warning = false; static Node *transformExprRecurse(ParseState *pstate, Node *expr); static Node *transformParamRef(ParseState *pstate, ParamRef *pref); @@ -95,6 +97,9 @@ static Expr *make_distinct_op(ParseState *pstate, List *opname, Node *ltree, Node *rtree, int location); static Node *make_nulltest_from_distinct(ParseState *pstate, A_Expr *distincta, Node *arg); +static Node *makeParamSessionVariable(ParseState *pstate, + Oid varid, Oid typid, int32 typmod, Oid collid, + char *attrname, int location); /* @@ -467,6 +472,88 @@ transformIndirection(ParseState *pstate, A_Indirection *ind) return result; } +/* + * Returns true if the given expression kind is valid for session variables + * Session variables can be used everywhere where external parameters can be + * used. Session variables are not allowed in DDL commands or in constraints. + * + * An identifier can be parsed as a session variable only for expression kinds + * where session variables are allowed. This is the primary usage of this + * function. + * + * Second usage of this function is to decide whether "column does not exist" or + * "column or variable does not exist" error message should be printed. + * When we are in an expression where session variables cannot be used, we raise + * the first form of error message. + */ +static bool +expr_kind_allows_session_variables(ParseExprKind p_expr_kind) +{ + bool result = false; + + switch (p_expr_kind) + { + case EXPR_KIND_NONE: + Assert(false); /* can't happen */ + return false; + + /* allow */ + case EXPR_KIND_OTHER: + case EXPR_KIND_JOIN_ON: + case EXPR_KIND_FROM_SUBSELECT: + case EXPR_KIND_FROM_FUNCTION: + case EXPR_KIND_WHERE: + case EXPR_KIND_HAVING: + case EXPR_KIND_FILTER: + case EXPR_KIND_WINDOW_PARTITION: + case EXPR_KIND_WINDOW_ORDER: + case EXPR_KIND_WINDOW_FRAME_RANGE: + case EXPR_KIND_WINDOW_FRAME_ROWS: + case EXPR_KIND_WINDOW_FRAME_GROUPS: + case EXPR_KIND_SELECT_TARGET: + case EXPR_KIND_INSERT_TARGET: + case EXPR_KIND_UPDATE_SOURCE: + case EXPR_KIND_UPDATE_TARGET: + case EXPR_KIND_MERGE_WHEN: + case EXPR_KIND_GROUP_BY: + case EXPR_KIND_ORDER_BY: + case EXPR_KIND_DISTINCT_ON: + case EXPR_KIND_LIMIT: + case EXPR_KIND_OFFSET: + case EXPR_KIND_RETURNING: + case EXPR_KIND_VALUES: + case EXPR_KIND_VALUES_SINGLE: + case EXPR_KIND_ALTER_COL_TRANSFORM: + case EXPR_KIND_EXECUTE_PARAMETER: + case EXPR_KIND_POLICY: + case EXPR_KIND_CALL_ARGUMENT: + case EXPR_KIND_COPY_WHERE: + case EXPR_KIND_LET_TARGET: + result = true; + break; + + /* not allow */ + case EXPR_KIND_CHECK_CONSTRAINT: + case EXPR_KIND_DOMAIN_CHECK: + case EXPR_KIND_COLUMN_DEFAULT: + case EXPR_KIND_FUNCTION_DEFAULT: + case EXPR_KIND_INDEX_EXPRESSION: + case EXPR_KIND_INDEX_PREDICATE: + case EXPR_KIND_STATS_EXPRESSION: + case EXPR_KIND_TRIGGER_WHEN: + case EXPR_KIND_PARTITION_BOUND: + case EXPR_KIND_PARTITION_EXPRESSION: + case EXPR_KIND_GENERATED_COLUMN: + case EXPR_KIND_VARIABLE_DEFAULT: + case EXPR_KIND_JOIN_USING: + case EXPR_KIND_CYCLE_MARK: + result = false; + break; + } + + return result; +} + /* * Transform a ColumnRef. * @@ -815,6 +902,104 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) parser_errposition(pstate, cref->location))); } + /* + * There are contexts where session's variables are not allowed. The + * question is if we want to identify session's variables in these + * contexts? The code can be more simple, when we don't do it, but then we + * cannot to raise maybe useful message like "you cannot to use session + * variables here". On second hand, in this case the warnings about + * session's variable shadowing can be messy. + */ + if (expr_kind_allows_session_variables(pstate->p_expr_kind)) + { + Oid varid = InvalidOid; + char *attrname = NULL; + bool not_unique; + + /* + * Session variables are shadowed by columns, routine's variables or + * routine's arguments ever. We don't want to use session variable + * when it is not exactly shadowed, but RTE is valid like: + * + * CREATE TYPE T AS (c int); CREATE VARIABLE foo AS T; CREATE TABLE + * foo(a int, b int); + * + * SELECT foo.a, foo.b, foo.c FROM foo; + * + * This case can be messy and then we disallow it. When we know, so + * possible variable will be shadowed, we try to identify variable + * only when session_variables_ambiguity_warning is requested. + */ + if (node || + (!node && relname && crerr == CRERR_NO_COLUMN)) + { + /* + * In this path we just try (if it is wanted) detect if session + * variable is shadowed. + */ + if (session_variables_ambiguity_warning) + { + /* + * The AccessShareLock is created on related session variable. The lock + * will be kept for the whole transaction. + */ + varid = IdentifyVariable(cref->fields, &attrname, ¬_unique, true); + + if (OidIsValid(varid)) + { + /* This path will ending by WARNING. Unlock variable first */ + UnlockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock); + + if (node) + ereport(WARNING, + (errcode(ERRCODE_AMBIGUOUS_COLUMN), + errmsg("session variable \"%s\" is shadowed", + NameListToString(cref->fields)), + errdetail("Session variables can be shadowed by columns, routine's variables and routine's arguments with the same name."), + parser_errposition(pstate, cref->location))); + else + /* session variable is shadowed by RTE */ + ereport(WARNING, + (errcode(ERRCODE_AMBIGUOUS_COLUMN), + errmsg("session variable \"%s.%s\" is shadowed", + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)), + errdetail("Session variables can be shadowed by tables or table's aliases with the same name."), + parser_errposition(pstate, cref->location))); + } + } + } + else + { + /* + * The AccessShareLock is created on related session variable. The lock + * will be kept for the whole transaction. + */ + varid = IdentifyVariable(cref->fields, &attrname, ¬_unique, false); + + if (OidIsValid(varid)) + { + Oid typid; + int32 typmod; + Oid collid; + + if (not_unique) + ereport(ERROR, + (errcode(ERRCODE_AMBIGUOUS_PARAMETER), + errmsg("session variable reference \"%s\" is ambiguous", + NameListToString(cref->fields)), + parser_errposition(pstate, cref->location))); + + get_session_variable_type_typmod_collid(varid, &typid, &typmod, + &collid); + + node = makeParamSessionVariable(pstate, + varid, typid, typmod, collid, + attrname, cref->location); + } + } + } + /* * Throw error if no translation found. */ @@ -849,6 +1034,72 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) return node; } +/* + * Generate param variable for reference to session variable + */ +static Node * +makeParamSessionVariable(ParseState *pstate, + Oid varid, Oid typid, int32 typmod, Oid collid, + char *attrname, int location) +{ + Param *param; + + param = makeNode(Param); + + param->paramkind = PARAM_VARIABLE; + param->paramvarid = varid; + param->paramtype = typid; + param->paramtypmod = typmod; + param->paramcollid = collid; + + pstate->p_hasSessionVariables = true; + + if (attrname != NULL) + { + TupleDesc tupdesc; + int i; + + tupdesc = lookup_rowtype_tupdesc_noerror(typid, typmod, true); + if (!tupdesc) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("variable \"%s.%s\" is of type \"%s\", which is not a composite type", + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid), + format_type_be(typid)), + parser_errposition(pstate, location))); + + for (i = 0; i < tupdesc->natts; i++) + { + Form_pg_attribute att = TupleDescAttr(tupdesc, i); + + if (strcmp(attrname, NameStr(att->attname)) == 0 && + !att->attisdropped) + { + /* Success, so generate a FieldSelect expression */ + FieldSelect *fselect = makeNode(FieldSelect); + + fselect->arg = (Expr *) param; + fselect->fieldnum = i + 1; + fselect->resulttype = att->atttypid; + fselect->resulttypmod = att->atttypmod; + /* save attribute's collation for parse_collate.c */ + fselect->resultcollid = att->attcollation; + + ReleaseTupleDesc(tupdesc); + return (Node *) fselect; + } + } + + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_COLUMN), + errmsg("could not identify column \"%s\" in variable", attrname), + parser_errposition(pstate, location))); + } + + return (Node *) param; +} + static Node * transformParamRef(ParseState *pstate, ParamRef *pref) { diff --git a/src/backend/tcop/pquery.c b/src/backend/tcop/pquery.c index 5565f200c3..5382c450dc 100644 --- a/src/backend/tcop/pquery.c +++ b/src/backend/tcop/pquery.c @@ -86,6 +86,9 @@ CreateQueryDesc(PlannedStmt *plannedstmt, qd->queryEnv = queryEnv; qd->instrument_options = instrument_options; /* instrumentation wanted? */ + qd->num_session_variables = 0; + qd->session_variables = NULL; + /* null these fields until set by ExecutorStart */ qd->tupDesc = NULL; qd->estate = NULL; diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c index e93d66a7ec..b2a345bda7 100644 --- a/src/backend/utils/adt/ruleutils.c +++ b/src/backend/utils/adt/ruleutils.c @@ -38,6 +38,7 @@ #include "catalog/pg_statistic_ext.h" #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "commands/defrem.h" #include "commands/tablespace.h" #include "common/keywords.h" @@ -515,6 +516,7 @@ static char *generate_function_name(Oid funcid, int nargs, static char *generate_operator_name(Oid operid, Oid arg1, Oid arg2); static void add_cast_to(StringInfo buf, Oid typid); static char *generate_qualified_type_name(Oid typid); +static char *generate_session_variable_name(Oid varid); static text *string_to_text(char *str); static char *flatten_reloptions(Oid relid); static void get_reloptions(StringInfo buf, Datum reloptions); @@ -8148,6 +8150,14 @@ get_parameter(Param *param, deparse_context *context) return; } + /* translate paramvarid to session variable name */ + if (param->paramkind == PARAM_VARIABLE) + { + appendStringInfo(context->buf, "%s", + generate_session_variable_name(param->paramvarid)); + return; + } + /* * If it's an external parameter, see if the outermost namespace provides * function argument names. @@ -12449,6 +12459,42 @@ generate_collation_name(Oid collid) return result; } +/* + * generate_session_variable_name + * Compute the name to display for a session variable specified by OID + * + * The result includes all necessary quoting and schema-prefixing. + */ +static char * +generate_session_variable_name(Oid varid) +{ + HeapTuple tup; + Form_pg_variable varform; + char *varname; + char *nspname; + char *result; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for variable %u", varid); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + varname = NameStr(varform->varname); + + if (!VariableIsVisible(varid)) + nspname = get_namespace_name_or_temp(varform->varnamespace); + else + nspname = NULL; + + result = quote_qualified_identifier(nspname, varname); + + ReleaseSysCache(tup); + + return result; +} + /* * Given a C string, produce a TEXT datum. * diff --git a/src/backend/utils/cache/plancache.c b/src/backend/utils/cache/plancache.c index 87210fcf62..a3bea4e951 100644 --- a/src/backend/utils/cache/plancache.c +++ b/src/backend/utils/cache/plancache.c @@ -58,6 +58,7 @@ #include "access/transam.h" #include "catalog/namespace.h" +#include "catalog/pg_variable.h" #include "executor/executor.h" #include "miscadmin.h" #include "nodes/nodeFuncs.h" @@ -1874,9 +1875,12 @@ ScanQueryForLocks(Query *parsetree, bool acquire) /* * Recurse into sublink subqueries, too. But we already did the ones in - * the rtable and cteList. + * the rtable and cteList. We need to force recursive call for session + * variables too, to find and lock variables used in query (see + * ScanQueryWalker). */ - if (parsetree->hasSubLinks) + if (parsetree->hasSubLinks || + parsetree->hasSessionVariables) { query_tree_walker(parsetree, ScanQueryWalker, (void *) &acquire, @@ -1885,7 +1889,8 @@ ScanQueryForLocks(Query *parsetree, bool acquire) } /* - * Walker to find sublink subqueries for ScanQueryForLocks + * Walker to find sublink subqueries or referenced session variables + * for ScanQueryForLocks */ static bool ScanQueryWalker(Node *node, bool *acquire) @@ -1900,6 +1905,20 @@ ScanQueryWalker(Node *node, bool *acquire) ScanQueryForLocks(castNode(Query, sub->subselect), *acquire); /* Fall through to process lefthand args of SubLink */ } + else if (IsA(node, Param)) + { + Param *p = (Param *) node; + + if (p->paramkind == PARAM_VARIABLE) + { + if (acquire) + LockDatabaseObject(VariableRelationId, p->paramvarid, + 0, AccessShareLock); + else + UnlockDatabaseObject(VariableRelationId, p->paramvarid, + 0, AccessShareLock); + } + } /* * Do NOT recurse into Query nodes, because ScanQueryForLocks already @@ -2031,7 +2050,9 @@ PlanCacheRelCallback(Datum arg, Oid relid) /* * PlanCacheObjectCallback - * Syscache inval callback function for PROCOID and TYPEOID caches + * Syscache inval callback function for TYPEOID, PROCOID, NAMESPACEOID, + * OPEROID, AMOPOPID, FOREIGNSERVEROID, FOREIGNDATAWRAPPEROID and VARIABLEOID + * caches. * * Invalidate all plans mentioning the object with the specified hash value, * or all plans mentioning any member of this cache if hashvalue == 0. diff --git a/src/backend/utils/fmgr/fmgr.c b/src/backend/utils/fmgr/fmgr.c index f72dd25efa..006bfc76d7 100644 --- a/src/backend/utils/fmgr/fmgr.c +++ b/src/backend/utils/fmgr/fmgr.c @@ -1998,9 +1998,13 @@ get_call_expr_arg_stable(Node *expr, int argnum) */ if (IsA(arg, Const)) return true; - if (IsA(arg, Param) && - ((Param *) arg)->paramkind == PARAM_EXTERN) - return true; + if (IsA(arg, Param)) + { + Param *p = (Param *) arg; + + if (p->paramkind == PARAM_EXTERN || p->paramkind == PARAM_VARIABLE) + return true; + } return false; } diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 8abf9bb644..37b8373e04 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -1539,6 +1539,16 @@ struct config_bool ConfigureNamesBool[] = false, NULL, NULL, NULL }, + { + {"session_variables_ambiguity_warning", PGC_USERSET, DEVELOPER_OPTIONS, + gettext_noop("Raise warning when reference to a session variable is ambiguous."), + NULL, + GUC_NOT_IN_SAMPLE + }, + &session_variables_ambiguity_warning, + false, + NULL, NULL, NULL + }, { {"db_user_namespace", PGC_SIGHUP, CONN_AUTH_AUTH, gettext_noop("Enables per-database user names."), diff --git a/src/include/catalog/namespace.h b/src/include/catalog/namespace.h index 787de15ed1..312ecfe5b3 100644 --- a/src/include/catalog/namespace.h +++ b/src/include/catalog/namespace.h @@ -168,6 +168,7 @@ extern void ResetTempTableNamespace(void); extern List *NamesFromList(List *names); extern Oid LookupVariable(const char *nspname, const char *varname, bool missing_ok); +extern Oid IdentifyVariable(List *names, char **attrname, bool *not_unique, bool noerror); extern OverrideSearchPath *GetOverrideSearchPath(MemoryContext context); extern OverrideSearchPath *CopyOverrideSearchPath(OverrideSearchPath *path); diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index e1034fdc49..9d47362cc7 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12035,7 +12035,6 @@ proname => 'brin_minmax_multi_summary_send', provolatile => 's', prorettype => 'bytea', proargtypes => 'pg_brin_minmax_multi_summary', prosrc => 'brin_minmax_multi_summary_send' }, - { oid => '8981', descr => 'arbitrary value from among input values', proname => 'any_value', prokind => 'a', proisstrict => 'f', prorettype => 'anyelement', proargtypes => 'anyelement', @@ -12043,5 +12042,11 @@ { oid => '8982', descr => 'aggregate transition function', proname => 'any_value_transfn', prorettype => 'anyelement', proargtypes => 'anyelement anyelement', prosrc => 'any_value_transfn' }, - +{ oid => '8488', descr => 'list of used session variables', + proname => 'pg_session_variables', prorows => '1000', proretset => 't', + provolatile => 's', prorettype => 'record', proargtypes => '', + proallargtypes => '{oid,text,text,oid,text,bool,bool,bool,bool}', + proargmodes => '{o,o,o,o,o,o,o,o,o}', + proargnames => '{varid,schema,name,typid,typname,removed,has_value,can_select,can_update}', + prosrc => 'pg_session_variables' }, ] diff --git a/src/include/commands/session_variable.h b/src/include/commands/session_variable.h index 343ee070a5..f442c5d3b8 100644 --- a/src/include/commands/session_variable.h +++ b/src/include/commands/session_variable.h @@ -1,7 +1,7 @@ /*------------------------------------------------------------------------- * - * sessionvariable.h - * prototypes for sessionvariable.c. + * session_variable.h + * prototypes for session_variable.c. * * * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group @@ -26,6 +26,14 @@ extern void SessionVariableCreatePostprocess(Oid varid, char eoxaction); extern void SessionVariableDropPostprocess(Oid varid); +extern Datum CopySessionVariable(Oid varid, bool *isNull, Oid *typid); +extern Datum CopySessionVariableWithTypeCheck(Oid varid, bool *isNull, Oid expected_typid); + +extern void SetSessionVariable(Oid varid, Datum value, bool isNull); +extern void SetSessionVariableWithSecurityCheck(Oid varid, Datum value, bool isNull); + +extern void ResetSessionVariables(void); + extern void AtPreEOXact_SessionVariable(bool isCommit); extern void AtEOSubXact_SessionVariable(bool isCommit, SubTransactionId mySubid, diff --git a/src/include/executor/execExpr.h b/src/include/executor/execExpr.h index 048573c2bc..7aed55e7c1 100644 --- a/src/include/executor/execExpr.h +++ b/src/include/executor/execExpr.h @@ -159,6 +159,7 @@ typedef enum ExprEvalOp EEOP_PARAM_EXEC, EEOP_PARAM_EXTERN, EEOP_PARAM_CALLBACK, + EEOP_PARAM_VARIABLE, /* return CaseTestExpr value */ EEOP_CASE_TESTVAL, @@ -385,6 +386,13 @@ typedef struct ExprEvalStep Oid paramtype; /* OID of parameter's datatype */ } param; + /* for EEOP_PARAM_VARIABLE */ + struct + { + Oid varid; /* OID of assigned variable */ + Oid vartype; /* OID of parameter's datatype */ + } vparam; + /* for EEOP_PARAM_CALLBACK */ struct { @@ -776,6 +784,8 @@ extern void ExecEvalParamExec(ExprState *state, ExprEvalStep *op, ExprContext *econtext); extern void ExecEvalParamExtern(ExprState *state, ExprEvalStep *op, ExprContext *econtext); +extern void ExecEvalParamVariable(ExprState *state, ExprEvalStep *op, + ExprContext *econtext); extern void ExecEvalSQLValueFunction(ExprState *state, ExprEvalStep *op); extern void ExecEvalCurrentOfExpr(ExprState *state, ExprEvalStep *op); extern void ExecEvalNextValueExpr(ExprState *state, ExprEvalStep *op); diff --git a/src/include/executor/execdesc.h b/src/include/executor/execdesc.h index af2bf36dfb..c4c6331774 100644 --- a/src/include/executor/execdesc.h +++ b/src/include/executor/execdesc.h @@ -48,6 +48,10 @@ typedef struct QueryDesc EState *estate; /* executor's query-wide state */ PlanState *planstate; /* tree of per-plan-node state */ + /* reference to session variables buffer */ + int num_session_variables; + SessionVariableValue *session_variables; + /* This field is set by ExecutorRun */ bool already_executed; /* true if previously executed */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 61b3517906..5a464c12e5 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -601,6 +601,18 @@ typedef struct AsyncRequest * tuples) */ } AsyncRequest; +/* ---------------- + * SessionVariableValue + * ---------------- + */ +typedef struct SessionVariableValue +{ + Oid varid; + Oid typid; + bool isnull; + Datum value; +} SessionVariableValue; + /* ---------------- * EState information * @@ -653,6 +665,13 @@ typedef struct EState ParamListInfo es_param_list_info; /* values of external params */ ParamExecData *es_param_exec_vals; /* values of internal params */ + /* Variables info: */ + /* number of used session variables */ + int es_num_session_variables; + + /* array of copied values of session variables */ + SessionVariableValue *es_session_variables; + QueryEnvironment *es_queryEnv; /* query environment */ /* Other working state: */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index eecbbd7768..1f067fc84f 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -167,6 +167,8 @@ typedef struct Query bool hasForUpdate pg_node_attr(query_jumble_ignore); /* rewriter has applied some RLS policy */ bool hasRowSecurity pg_node_attr(query_jumble_ignore); + /* uses session variables */ + bool hasSessionVariables pg_node_attr(query_jumble_ignore); /* is a RETURN statement */ bool isReturn pg_node_attr(query_jumble_ignore); diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 7d4f24d250..4f698556a2 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -160,6 +160,9 @@ typedef struct PlannerGlobal /* partition descriptors */ PartitionDirectory partition_directory pg_node_attr(read_write_ignore); + + /* list of used session variables */ + List *sessionVariables; } PlannerGlobal; /* macro for fetching the Plan associated with a SubPlan node */ @@ -499,6 +502,8 @@ struct PlannerInfo bool placeholdersFrozen; /* true if planning a recursive WITH item */ bool hasRecursion; + /* true if session variables were used */ + bool hasSessionVariables; /* * Information about aggregates. Filled by preprocess_aggrefs(). diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index 1b787fe031..d287a80dce 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -95,6 +95,8 @@ typedef struct PlannedStmt Node *utilityStmt; /* non-null if this is utility stmt */ + List *sessionVariables; /* OIDs for PARAM_VARIABLE Params */ + /* statement location in source string (copied from Query) */ int stmt_location; /* start location, or -1 if unknown */ int stmt_len; /* length in bytes; 0 means "rest of string" */ diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h index 08e7dae73f..d27038a240 100644 --- a/src/include/nodes/primnodes.h +++ b/src/include/nodes/primnodes.h @@ -43,7 +43,9 @@ typedef struct Alias List *colnames; /* optional list of column aliases */ } Alias; -/* What to do at commit time for temporary relations */ +/* + * What to do at commit time for temporary relations or session variables. + */ typedef enum OnCommitAction { ONCOMMIT_NOOP, /* No ON COMMIT clause (do nothing) */ @@ -339,13 +341,17 @@ typedef struct Const * of the `paramid' field contain the SubLink's subLinkId, and * the low-order 16 bits contain the column number. (This type * of Param is also converted to PARAM_EXEC during planning.) + * + * PARAM_VARIABLE: The parameter is an access to session variable + * paramid holds varid. */ typedef enum ParamKind { PARAM_EXTERN, PARAM_EXEC, PARAM_SUBLINK, - PARAM_MULTIEXPR + PARAM_MULTIEXPR, + PARAM_VARIABLE } ParamKind; typedef struct Param @@ -358,6 +364,8 @@ typedef struct Param int32 paramtypmod pg_node_attr(query_jumble_ignore); /* OID of collation, or InvalidOid if none */ Oid paramcollid pg_node_attr(query_jumble_ignore); + /* OID of session variable if it is used */ + Oid paramvarid; /* token location, or -1 if unknown */ int location; } Param; diff --git a/src/include/optimizer/planmain.h b/src/include/optimizer/planmain.h index 5fc900737d..f106768beb 100644 --- a/src/include/optimizer/planmain.h +++ b/src/include/optimizer/planmain.h @@ -114,4 +114,6 @@ extern void record_plan_function_dependency(PlannerInfo *root, Oid funcid); extern void record_plan_type_dependency(PlannerInfo *root, Oid typid); extern bool extract_query_dependencies_walker(Node *node, PlannerInfo *context); +extern void pull_up_has_session_variables(PlannerInfo *root); + #endif /* PLANMAIN_H */ diff --git a/src/include/parser/parse_expr.h b/src/include/parser/parse_expr.h index 7d38ca75f7..f6a5ccb0c1 100644 --- a/src/include/parser/parse_expr.h +++ b/src/include/parser/parse_expr.h @@ -17,6 +17,7 @@ /* GUC parameters */ extern PGDLLIMPORT bool Transform_null_equals; +extern PGDLLIMPORT bool session_variables_ambiguity_warning; extern Node *transformExpr(ParseState *pstate, Node *expr, ParseExprKind exprKind); diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h index eccde83628..4e9692d05a 100644 --- a/src/include/parser/parse_node.h +++ b/src/include/parser/parse_node.h @@ -225,6 +225,7 @@ struct ParseState bool p_hasTargetSRFs; bool p_hasSubLinks; bool p_hasModifyingCTE; + bool p_hasSessionVariables; Node *p_last_srf; /* most recent set-returning func/op found */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 8c1df4b1fd..69671f2c78 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2483,6 +2483,7 @@ SerializedTransactionState Session SessionBackupState SessionEndType +SessionVariableValue SetConstraintState SetConstraintStateData SetConstraintTriggerData @@ -2667,6 +2668,8 @@ SupportRequestRows SupportRequestSelectivity SupportRequestSimplify SupportRequestWFuncMonotonic +SVariable +SVariableData SVariableXActAction SVariableXActActionItem Syn -- 2.40.1 [text/x-patch] v20230517-1-0001-catalog-support-for-session-variables.patch (85.2K, ../../CAFj8pRAM60yANCnqKHHXZM3j++wpXWOu+rw6tVq5m531LHXiSw@mail.gmail.com/12-v20230517-1-0001-catalog-support-for-session-variables.patch) download | inline diff: From d63f8abe3b30a518ca4f920bf901fbbfbfa27135 Mon Sep 17 00:00:00 2001 From: "[email protected]" <[email protected]> Date: Sun, 13 Nov 2022 17:39:37 +0100 Subject: [PATCH 01/10] catalog support for session variables Implementation new system object class - session variable with new access rights SELECT, UPDATE, with routines for creating session variable, initialization of session variable from system catalog, and lookups routines for identification of session variables. --- src/backend/access/transam/xact.c | 11 + src/backend/catalog/Makefile | 4 +- src/backend/catalog/aclchk.c | 77 +++++ src/backend/catalog/dependency.c | 13 +- src/backend/catalog/meson.build | 1 + src/backend/catalog/namespace.c | 140 +++++++++ src/backend/catalog/objectaddress.c | 122 +++++++- src/backend/catalog/pg_shdepend.c | 2 + src/backend/catalog/pg_variable.c | 378 ++++++++++++++++++++++++ src/backend/commands/Makefile | 1 + src/backend/commands/alter.c | 9 + src/backend/commands/dropcmds.c | 4 + src/backend/commands/event_trigger.c | 6 + src/backend/commands/seclabel.c | 1 + src/backend/commands/session_variable.c | 243 +++++++++++++++ src/backend/commands/tablecmds.c | 43 +++ src/backend/parser/gram.y | 145 ++++++++- src/backend/parser/parse_agg.c | 2 + src/backend/parser/parse_expr.c | 5 + src/backend/parser/parse_func.c | 1 + src/backend/parser/parse_utilcmd.c | 12 + src/backend/tcop/utility.c | 16 + src/backend/utils/adt/acl.c | 7 + src/backend/utils/cache/lsyscache.c | 113 +++++++ src/backend/utils/cache/syscache.c | 14 + src/include/catalog/dependency.h | 5 +- src/include/catalog/meson.build | 1 + src/include/catalog/namespace.h | 5 + src/include/catalog/pg_default_acl.h | 1 + src/include/catalog/pg_proc.dat | 3 + src/include/catalog/pg_variable.h | 130 ++++++++ src/include/commands/session_variable.h | 34 +++ src/include/nodes/parsenodes.h | 20 ++ src/include/parser/kwlist.h | 2 + src/include/parser/parse_node.h | 1 + src/include/tcop/cmdtaglist.h | 3 + src/include/utils/acl.h | 1 + src/include/utils/lsyscache.h | 9 + src/include/utils/syscache.h | 6 +- src/test/regress/expected/oidjoins.out | 4 + src/tools/pgindent/typedefs.list | 5 + 41 files changed, 1588 insertions(+), 12 deletions(-) create mode 100644 src/backend/catalog/pg_variable.c create mode 100644 src/backend/commands/session_variable.c create mode 100644 src/include/catalog/pg_variable.h create mode 100644 src/include/commands/session_variable.h diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index 6a837e1539..ee20c2e520 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -36,6 +36,7 @@ #include "catalog/pg_enum.h" #include "catalog/storage.h" #include "commands/async.h" +#include "commands/session_variable.h" #include "commands/tablecmds.h" #include "commands/trigger.h" #include "common/pg_prng.h" @@ -2224,6 +2225,9 @@ CommitTransaction(void) */ smgrDoPendingSyncs(true, is_parallel_worker); + /* Let ON COMMIT DROP */ + AtPreEOXact_SessionVariable(true); + /* close large objects before lower-level cleanup */ AtEOXact_LargeObject(true); @@ -2800,6 +2804,9 @@ AbortTransaction(void) AtAbort_Portals(); smgrDoPendingSyncs(false, is_parallel_worker); AtEOXact_LargeObject(false); + + /* 'false' means it's abort */ + AtPreEOXact_SessionVariable(false); AtAbort_Notify(); AtEOXact_RelationMap(false, is_parallel_worker); AtAbort_Twophase(); @@ -5024,6 +5031,8 @@ CommitSubTransaction(void) AtEOSubXact_SPI(true, s->subTransactionId); AtEOSubXact_on_commit_actions(true, s->subTransactionId, s->parent->subTransactionId); + AtEOSubXact_SessionVariable(true, s->subTransactionId, + s->parent->subTransactionId); AtEOSubXact_Namespace(true, s->subTransactionId, s->parent->subTransactionId); AtEOSubXact_Files(true, s->subTransactionId, @@ -5187,6 +5196,8 @@ AbortSubTransaction(void) AtEOSubXact_SPI(false, s->subTransactionId); AtEOSubXact_on_commit_actions(false, s->subTransactionId, s->parent->subTransactionId); + AtEOSubXact_SessionVariable(false, s->subTransactionId, + s->parent->subTransactionId); AtEOSubXact_Namespace(false, s->subTransactionId, s->parent->subTransactionId); AtEOSubXact_Files(false, s->subTransactionId, diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile index a60107bf94..34a8ac98a8 100644 --- a/src/backend/catalog/Makefile +++ b/src/backend/catalog/Makefile @@ -45,6 +45,7 @@ OBJS = \ pg_shdepend.o \ pg_subscription.o \ pg_type.o \ + pg_variable.o \ storage.o \ toasting.o @@ -72,7 +73,8 @@ CATALOG_HEADERS := \ pg_collation.h pg_parameter_acl.h pg_partitioned_table.h \ pg_range.h pg_transform.h \ pg_sequence.h pg_publication.h pg_publication_namespace.h \ - pg_publication_rel.h pg_subscription.h pg_subscription_rel.h + pg_publication_rel.h pg_subscription.h pg_subscription_rel.h \ + pg_variable.h GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h) schemapg.h system_fk_info.h diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c index 45cdcd3dc6..c2f6ca4a1d 100644 --- a/src/backend/catalog/aclchk.c +++ b/src/backend/catalog/aclchk.c @@ -60,6 +60,7 @@ #include "catalog/pg_ts_parser.h" #include "catalog/pg_ts_template.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "commands/dbcommands.h" #include "commands/defrem.h" #include "commands/event_trigger.h" @@ -281,6 +282,9 @@ restrict_and_check_grant(bool is_grant, AclMode avail_goptions, bool all_privs, case OBJECT_PARAMETER_ACL: whole_mask = ACL_ALL_RIGHTS_PARAMETER_ACL; break; + case OBJECT_VARIABLE: + whole_mask = ACL_ALL_RIGHTS_VARIABLE; + break; default: elog(ERROR, "unrecognized object type: %d", objtype); /* not reached, but keep compiler quiet */ @@ -525,6 +529,10 @@ ExecuteGrantStmt(GrantStmt *stmt) all_privileges = ACL_ALL_RIGHTS_PARAMETER_ACL; errormsg = gettext_noop("invalid privilege type %s for parameter"); break; + case OBJECT_VARIABLE: + all_privileges = ACL_ALL_RIGHTS_VARIABLE; + errormsg = gettext_noop("invalid privilege type %s for session variable"); + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) stmt->objtype); @@ -630,6 +638,9 @@ ExecGrantStmt_oids(InternalGrant *istmt) case OBJECT_PARAMETER_ACL: ExecGrant_Parameter(istmt); break; + case OBJECT_VARIABLE: + ExecGrant_common(istmt, VariableRelationId, ACL_ALL_RIGHTS_VARIABLE, NULL); + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) istmt->objtype); @@ -820,6 +831,18 @@ objectNamesToOids(ObjectType objtype, List *objnames, bool is_grant) objects = lappend_oid(objects, parameterId); } break; + case OBJECT_VARIABLE: + foreach(cell, objnames) + { + RangeVar *varvar = (RangeVar *) lfirst(cell); + Oid relOid; + + relOid = LookupVariable(varvar->schemaname, + varvar->relname, + false); + objects = lappend_oid(objects, relOid); + } + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) objtype); @@ -909,6 +932,32 @@ objectsInSchemaToOids(ObjectType objtype, List *nspnames) table_close(rel, AccessShareLock); } break; + case OBJECT_VARIABLE: + { + ScanKeyData key; + Relation rel; + TableScanDesc scan; + HeapTuple tuple; + + ScanKeyInit(&key, + Anum_pg_variable_varnamespace, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(namespaceId)); + + rel = table_open(VariableRelationId, AccessShareLock); + scan = table_beginscan_catalog(rel, 1, &key); + + while ((tuple = heap_getnext(scan, ForwardScanDirection)) != NULL) + { + Oid oid = ((Form_pg_proc) GETSTRUCT(tuple))->oid; + + objects = lappend_oid(objects, oid); + } + + table_endscan(scan); + table_close(rel, AccessShareLock); + } + break; default: /* should not happen */ elog(ERROR, "unrecognized GrantStmt.objtype: %d", @@ -1068,6 +1117,10 @@ ExecAlterDefaultPrivilegesStmt(ParseState *pstate, AlterDefaultPrivilegesStmt *s all_privileges = ACL_ALL_RIGHTS_SCHEMA; errormsg = gettext_noop("invalid privilege type %s for schema"); break; + case OBJECT_VARIABLE: + all_privileges = ACL_ALL_RIGHTS_VARIABLE; + errormsg = gettext_noop("invalid privilege type %s for session variable"); + break; default: elog(ERROR, "unrecognized GrantStmt.objtype: %d", (int) action->objtype); @@ -1259,6 +1312,12 @@ SetDefaultACL(InternalDefaultACL *iacls) this_privileges = ACL_ALL_RIGHTS_SCHEMA; break; + case OBJECT_VARIABLE: + objtype = DEFACLOBJ_VARIABLE; + if (iacls->all_privs && this_privileges == ACL_NO_RIGHTS) + this_privileges = ACL_ALL_RIGHTS_VARIABLE; + break; + default: elog(ERROR, "unrecognized object type: %d", (int) iacls->objtype); @@ -1490,6 +1549,9 @@ RemoveRoleFromObjectACL(Oid roleid, Oid classid, Oid objid) case DEFACLOBJ_NAMESPACE: iacls.objtype = OBJECT_SCHEMA; break; + case DEFACLOBJ_VARIABLE: + iacls.objtype = OBJECT_VARIABLE; + break; default: /* Shouldn't get here */ elog(ERROR, "unexpected default ACL type: %d", @@ -1550,6 +1612,9 @@ RemoveRoleFromObjectACL(Oid roleid, Oid classid, Oid objid) case ParameterAclRelationId: istmt.objtype = OBJECT_PARAMETER_ACL; break; + case VariableRelationId: + istmt.objtype = OBJECT_VARIABLE; + break; default: elog(ERROR, "unexpected object class %u", classid); break; @@ -2783,6 +2848,9 @@ aclcheck_error(AclResult aclerr, ObjectType objtype, case OBJECT_TYPE: msg = gettext_noop("permission denied for type %s"); break; + case OBJECT_VARIABLE: + msg = gettext_noop("permission denied for session variable %s"); + break; case OBJECT_VIEW: msg = gettext_noop("permission denied for view %s"); break; @@ -2894,6 +2962,9 @@ aclcheck_error(AclResult aclerr, ObjectType objtype, case OBJECT_TYPE: msg = gettext_noop("must be owner of type %s"); break; + case OBJECT_VARIABLE: + msg = gettext_noop("must be owner of session variable %s"); + break; case OBJECT_VIEW: msg = gettext_noop("must be owner of view %s"); break; @@ -3042,6 +3113,8 @@ pg_aclmask(ObjectType objtype, Oid object_oid, AttrNumber attnum, Oid roleid, return ACL_NO_RIGHTS; case OBJECT_TYPE: return object_aclmask(TypeRelationId, object_oid, roleid, mask, how); + case OBJECT_VARIABLE: + return object_aclmask(VariableRelationId, object_oid, roleid, mask, how); default: elog(ERROR, "unrecognized object type: %d", (int) objtype); @@ -4166,6 +4239,10 @@ get_user_default_acl(ObjectType objtype, Oid ownerId, Oid nsp_oid) defaclobjtype = DEFACLOBJ_NAMESPACE; break; + case OBJECT_VARIABLE: + defaclobjtype = DEFACLOBJ_VARIABLE; + break; + default: return NULL; } diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index f8a136ba0a..77acbeda80 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -65,12 +65,15 @@ #include "catalog/pg_ts_template.h" #include "catalog/pg_type.h" #include "catalog/pg_user_mapping.h" +#include "catalog/pg_variable.h" #include "commands/comment.h" #include "commands/defrem.h" #include "commands/event_trigger.h" #include "commands/extension.h" #include "commands/policy.h" #include "commands/publicationcmds.h" +#include "commands/schemacmds.h" +#include "commands/session_variable.h" #include "commands/seclabel.h" #include "commands/sequence.h" #include "commands/trigger.h" @@ -188,7 +191,8 @@ static const Oid object_classes[] = { PublicationRelationId, /* OCLASS_PUBLICATION */ PublicationRelRelationId, /* OCLASS_PUBLICATION_REL */ SubscriptionRelationId, /* OCLASS_SUBSCRIPTION */ - TransformRelationId /* OCLASS_TRANSFORM */ + TransformRelationId, /* OCLASS_TRANSFORM */ + VariableRelationId /* OCLASS_VARIABLE */ }; /* @@ -1514,6 +1518,10 @@ doDeletion(const ObjectAddress *object, int flags) DropObjectById(object); break; + case OCLASS_VARIABLE: + DropVariable(object->objectId); + break; + /* * These global object types are not supported here. */ @@ -2966,6 +2974,9 @@ getObjectClass(const ObjectAddress *object) case TransformRelationId: return OCLASS_TRANSFORM; + + case VariableRelationId: + return OCLASS_VARIABLE; } /* shouldn't get here */ diff --git a/src/backend/catalog/meson.build b/src/backend/catalog/meson.build index fa6609e577..f48e3cb07d 100644 --- a/src/backend/catalog/meson.build +++ b/src/backend/catalog/meson.build @@ -32,6 +32,7 @@ backend_sources += files( 'pg_shdepend.c', 'pg_subscription.c', 'pg_type.c', + 'pg_variable.c', 'storage.c', 'toasting.c', ) diff --git a/src/backend/catalog/namespace.c b/src/backend/catalog/namespace.c index 73ddb67882..9a04530d1e 100644 --- a/src/backend/catalog/namespace.c +++ b/src/backend/catalog/namespace.c @@ -40,6 +40,7 @@ #include "catalog/pg_ts_parser.h" #include "catalog/pg_ts_template.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "commands/dbcommands.h" #include "funcapi.h" #include "mb/pg_wchar.h" @@ -765,6 +766,69 @@ RelationIsVisible(Oid relid) return visible; } +/* + * VariableIsVisible + * Determine whether a variable (identified by OID) is visible in the + * current search path. Visible means "would be found by searching + * for the unqualified variable name". + */ +bool +VariableIsVisible(Oid varid) +{ + HeapTuple vartup; + Form_pg_variable varform; + Oid varnamespace; + bool visible; + + vartup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + if (!HeapTupleIsValid(vartup)) + elog(ERROR, "cache lookup failed for session variable %u", varid); + varform = (Form_pg_variable) GETSTRUCT(vartup); + + recomputeNamespacePath(); + + /* + * Quick check: if it ain't in the path at all, it ain't visible. Items in + * the system namespace are surely in the path and so we needn't even do + * list_member_oid() for them. + */ + varnamespace = varform->varnamespace; + if (varnamespace != PG_CATALOG_NAMESPACE && + !list_member_oid(activeSearchPath, varnamespace)) + visible = false; + else + { + /* + * If it is in the path, it might still not be visible; it could be + * hidden by another variable of the same name earlier in the path. So + * we must do a slow check for conflicting relations. + */ + char *varname = NameStr(varform->varname); + ListCell *l; + + visible = false; + foreach(l, activeSearchPath) + { + Oid namespaceId = lfirst_oid(l); + + if (namespaceId == varnamespace) + { + /* Found it first in path */ + visible = true; + break; + } + if (OidIsValid(get_varname_varid(varname, namespaceId))) + { + /* Found something else first in path */ + break; + } + } + } + + ReleaseSysCache(vartup); + + return visible; +} /* * TypenameGetTypid @@ -2840,6 +2904,71 @@ TSConfigIsVisible(Oid cfgid) return visible; } +/* + * Returns oid of session variable specified by possibly qualified identifier. + * + * If not found, returns InvalidOid if missing_ok, else throws error. + * When rowtype_only argument is true the session variables of not + * composite types are ignored. This should to reduce possible collisions. + */ +Oid +LookupVariable(const char *nspname, + const char *varname, + bool missing_ok) +{ + Oid namespaceId; + Oid varoid = InvalidOid; + ListCell *l; + + if (nspname) + { + namespaceId = LookupExplicitNamespace(nspname, missing_ok); + + /* + * If nspname is not a known namespace, then nspname.varname cannot be + * any usable session variable. + */ + if (OidIsValid(namespaceId)) + { + varoid = GetSysCacheOid2(VARIABLENAMENSP, Anum_pg_variable_oid, + PointerGetDatum(varname), + ObjectIdGetDatum(namespaceId)); + } + } + else + { + /* Iterate over schemas in search_path */ + recomputeNamespacePath(); + + foreach(l, activeSearchPath) + { + namespaceId = lfirst_oid(l); + + varoid = GetSysCacheOid2(VARIABLENAMENSP, Anum_pg_variable_oid, + PointerGetDatum(varname), + ObjectIdGetDatum(namespaceId)); + + if (OidIsValid(varoid)) + break; + } + } + + if (!OidIsValid(varoid) && !missing_ok) + { + if (nspname) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("session variable \"%s.%s\" does not exist", + nspname, varname))); + else + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("session variable \"%s\" does not exist", + varname))); + } + + return varoid; +} /* * DeconstructQualifiedName @@ -4661,3 +4790,14 @@ pg_is_other_temp_schema(PG_FUNCTION_ARGS) PG_RETURN_BOOL(isOtherTempNamespace(oid)); } + +Datum +pg_variable_is_visible(PG_FUNCTION_ARGS) +{ + Oid oid = PG_GETARG_OID(0); + + if (!SearchSysCacheExists1(VARIABLEOID, ObjectIdGetDatum(oid))) + PG_RETURN_NULL(); + + PG_RETURN_BOOL(VariableIsVisible(oid)); +} diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c index 95fefc7565..d5930f2627 100644 --- a/src/backend/catalog/objectaddress.c +++ b/src/backend/catalog/objectaddress.c @@ -64,6 +64,7 @@ #include "catalog/pg_ts_template.h" #include "catalog/pg_type.h" #include "catalog/pg_user_mapping.h" +#include "catalog/pg_variable.h" #include "commands/dbcommands.h" #include "commands/defrem.h" #include "commands/event_trigger.h" @@ -633,6 +634,20 @@ static const ObjectPropertyType ObjectProperty[] = OBJECT_USER_MAPPING, false }, + { + "session variable", + VariableRelationId, + VariableObjectIndexId, + VARIABLEOID, + VARIABLENAMENSP, + Anum_pg_variable_oid, + Anum_pg_variable_varname, + Anum_pg_variable_varnamespace, + Anum_pg_variable_varowner, + Anum_pg_variable_varacl, + OBJECT_VARIABLE, + true + } }; /* @@ -869,6 +884,10 @@ static const struct object_type_map /* OCLASS_STATISTIC_EXT */ { "statistics object", OBJECT_STATISTIC_EXT + }, + /* OCLASS_VARIABLE */ + { + "session variable", OBJECT_VARIABLE } }; @@ -894,6 +913,7 @@ static ObjectAddress get_object_address_attrdef(ObjectType objtype, bool missing_ok); static ObjectAddress get_object_address_type(ObjectType objtype, TypeName *typename, bool missing_ok); +static ObjectAddress get_object_address_variable(List *object, bool missing_ok); static ObjectAddress get_object_address_opcf(ObjectType objtype, List *object, bool missing_ok); static ObjectAddress get_object_address_opf_member(ObjectType objtype, @@ -1164,6 +1184,9 @@ get_object_address(ObjectType objtype, Node *object, missing_ok); address.objectSubId = 0; break; + case OBJECT_VARIABLE: + address = get_object_address_variable(castNode(List, object), missing_ok); + break; /* no default, to let compiler warn about missing case */ } @@ -2038,16 +2061,20 @@ get_object_address_defacl(List *object, bool missing_ok) case DEFACLOBJ_NAMESPACE: objtype_str = "schemas"; break; + case DEFACLOBJ_VARIABLE: + objtype_str = "variables"; + break; default: ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("unrecognized default ACL object type \"%c\"", objtype), - errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", + errhint("Valid object types are \"%c\", \"%c\", \"%c\", \"%c\", \"%c\", \"%c\".", DEFACLOBJ_RELATION, DEFACLOBJ_SEQUENCE, DEFACLOBJ_FUNCTION, DEFACLOBJ_TYPE, - DEFACLOBJ_NAMESPACE))); + DEFACLOBJ_NAMESPACE, + DEFACLOBJ_VARIABLE))); } /* @@ -2131,6 +2158,24 @@ textarray_to_strvaluelist(ArrayType *arr) return list; } +/* + * Find the ObjectAddress for a session variable + */ +static ObjectAddress +get_object_address_variable(List *object, bool missing_ok) +{ + ObjectAddress address; + char *nspname = NULL; + char *varname = NULL; + + ObjectAddressSet(address, VariableRelationId, InvalidOid); + + DeconstructQualifiedName(object, &nspname, &varname); + address.objectId = LookupVariable(nspname, varname, missing_ok); + + return address; +} + /* * SQL-callable version of get_object_address */ @@ -2325,6 +2370,7 @@ pg_get_object_address(PG_FUNCTION_ARGS) case OBJECT_TABCONSTRAINT: case OBJECT_OPCLASS: case OBJECT_OPFAMILY: + case OBJECT_VARIABLE: objnode = (Node *) name; break; case OBJECT_ACCESS_METHOD: @@ -2496,6 +2542,7 @@ check_object_ownership(Oid roleid, ObjectType objtype, ObjectAddress address, case OBJECT_STATISTIC_EXT: case OBJECT_TSDICTIONARY: case OBJECT_TSCONFIGURATION: + case OBJECT_VARIABLE: if (!object_ownercheck(address.classId, address.objectId, roleid)) aclcheck_error(ACLCHECK_NOT_OWNER, objtype, NameListToString(castNode(List, object))); @@ -3484,6 +3531,32 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok) break; } + case OCLASS_VARIABLE: + { + char *nspname; + HeapTuple tup; + Form_pg_variable varform; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(object->objectId)); + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for session variable %u", + object->objectId); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + if (VariableIsVisible(object->objectId)) + nspname = NULL; + else + nspname = get_namespace_name(varform->varnamespace); + + appendStringInfo(&buffer, _("session variable %s"), + quote_qualified_identifier(nspname, + NameStr(varform->varname))); + + ReleaseSysCache(tup); + break; + } + case OCLASS_TSPARSER: { HeapTuple tup; @@ -3836,6 +3909,16 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok) _("default privileges on new schemas belonging to role %s"), rolename); break; + case DEFACLOBJ_VARIABLE: + if (nspname) + appendStringInfo(&buffer, + _("default privileges on new session variables belonging to role %s in schema %s"), + rolename, nspname); + else + appendStringInfo(&buffer, + _("default privileges on new session variables belonging to role %s"), + rolename); + break; default: /* shouldn't get here */ if (nspname) @@ -4585,6 +4668,10 @@ getObjectTypeDescription(const ObjectAddress *object, bool missing_ok) appendStringInfoString(&buffer, "transform"); break; + case OCLASS_VARIABLE: + appendStringInfoString(&buffer, "session variable"); + break; + /* * There's intentionally no default: case here; we want the * compiler to warn if a new OCLASS hasn't been handled above. @@ -5692,6 +5779,10 @@ getObjectIdentityParts(const ObjectAddress *object, appendStringInfoString(&buffer, " on schemas"); break; + case DEFACLOBJ_VARIABLE: + appendStringInfoString(&buffer, + " on session variables"); + break; } if (objname) @@ -5932,6 +6023,33 @@ getObjectIdentityParts(const ObjectAddress *object, } break; + case OCLASS_VARIABLE: + { + char *schema; + char *varname; + HeapTuple tup; + Form_pg_variable varform; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(object->objectId)); + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for session variable %u", + object->objectId); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + schema = get_namespace_name_or_temp(varform->varnamespace); + varname = NameStr(varform->varname); + + appendStringInfo(&buffer, "%s", + quote_qualified_identifier(schema, varname)); + + if (objname) + *objname = list_make2(schema, varname); + + ReleaseSysCache(tup); + break; + } + /* * There's intentionally no default: case here; we want the * compiler to warn if a new OCLASS hasn't been handled above. diff --git a/src/backend/catalog/pg_shdepend.c b/src/backend/catalog/pg_shdepend.c index 64d326f073..848b36a87e 100644 --- a/src/backend/catalog/pg_shdepend.c +++ b/src/backend/catalog/pg_shdepend.c @@ -47,6 +47,7 @@ #include "catalog/pg_ts_dict.h" #include "catalog/pg_type.h" #include "catalog/pg_user_mapping.h" +#include "catalog/pg_variable.h" #include "commands/alter.h" #include "commands/collationcmds.h" #include "commands/conversioncmds.h" @@ -1613,6 +1614,7 @@ shdepReassignOwned(List *roleids, Oid newrole) case DatabaseRelationId: case TSConfigRelationId: case TSDictionaryRelationId: + case VariableRelationId: { Oid classId = sdepForm->classid; Relation catalog; diff --git a/src/backend/catalog/pg_variable.c b/src/backend/catalog/pg_variable.c new file mode 100644 index 0000000000..462126bf7a --- /dev/null +++ b/src/backend/catalog/pg_variable.c @@ -0,0 +1,378 @@ +/*------------------------------------------------------------------------- + * + * pg_variable.c + * session variables + * + * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/backend/catalog/pg_variable.c + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/heapam.h" +#include "access/htup_details.h" +#include "catalog/catalog.h" +#include "catalog/dependency.h" +#include "catalog/indexing.h" +#include "catalog/namespace.h" +#include "catalog/objectaccess.h" +#include "catalog/pg_collation.h" +#include "catalog/pg_namespace.h" +#include "catalog/pg_type.h" +#include "catalog/pg_variable.h" +#include "commands/session_variable.h" +#include "miscadmin.h" +#include "parser/parse_coerce.h" +#include "parser/parse_collate.h" +#include "parser/parse_expr.h" +#include "parser/parse_type.h" +#include "storage/lmgr.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/pg_lsn.h" +#include "utils/syscache.h" + + +static ObjectAddress create_variable(const char *varName, + Oid varNamespace, + Oid varType, + int32 varTypmod, + Oid varOwner, + Oid varCollation, + Node *varDefexpr, + VariableEOXAction eoxaction, + bool is_not_null, + bool if_not_exists, + bool is_immutable); + + +/* + * Creates entry in pg_variable table + */ +static ObjectAddress +create_variable(const char *varName, + Oid varNamespace, + Oid varType, + int32 varTypmod, + Oid varOwner, + Oid varCollation, + Node *varDefexpr, + VariableEOXAction eoxaction, + bool is_not_null, + bool if_not_exists, + bool is_immutable) +{ + Acl *varacl; + NameData varname; + bool nulls[Natts_pg_variable]; + Datum values[Natts_pg_variable]; + Relation rel; + HeapTuple tup; + TupleDesc tupdesc; + ObjectAddress myself, + referenced; + ObjectAddresses *addrs; + Oid varid; + + Assert(varName); + Assert(OidIsValid(varNamespace)); + Assert(OidIsValid(varType)); + Assert(OidIsValid(varOwner)); + + rel = table_open(VariableRelationId, RowExclusiveLock); + + /* + * Check for duplicates. Note that this does not really prevent + * duplicates, it's here just to provide nicer error message in common + * case. The real protection is the unique key on the catalog. + */ + if (SearchSysCacheExists2(VARIABLENAMENSP, + PointerGetDatum(varName), + ObjectIdGetDatum(varNamespace))) + { + if (if_not_exists) + ereport(NOTICE, + (errcode(ERRCODE_DUPLICATE_OBJECT), + errmsg("session variable \"%s\" already exists, skipping", + varName))); + else + ereport(ERROR, + (errcode(ERRCODE_DUPLICATE_OBJECT), + errmsg("session variable \"%s\" already exists", + varName))); + + table_close(rel, RowExclusiveLock); + + return InvalidObjectAddress; + } + + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + + namestrcpy(&varname, varName); + + varid = GetNewOidWithIndex(rel, VariableObjectIndexId, Anum_pg_variable_oid); + + values[Anum_pg_variable_oid - 1] = ObjectIdGetDatum(varid); + values[Anum_pg_variable_varcreate_lsn - 1] = LSNGetDatum(GetXLogInsertRecPtr()); + values[Anum_pg_variable_varname - 1] = NameGetDatum(&varname); + values[Anum_pg_variable_varnamespace - 1] = ObjectIdGetDatum(varNamespace); + values[Anum_pg_variable_vartype - 1] = ObjectIdGetDatum(varType); + values[Anum_pg_variable_vartypmod - 1] = Int32GetDatum(varTypmod); + values[Anum_pg_variable_varowner - 1] = ObjectIdGetDatum(varOwner); + values[Anum_pg_variable_varcollation - 1] = ObjectIdGetDatum(varCollation); + values[Anum_pg_variable_varisnotnull - 1] = BoolGetDatum(is_not_null); + values[Anum_pg_variable_varisimmutable - 1] = BoolGetDatum(is_immutable); + values[Anum_pg_variable_vareoxaction - 1] = CharGetDatum(eoxaction); + + /* varacl will be determined later */ + + if (varDefexpr) + values[Anum_pg_variable_vardefexpr - 1] = CStringGetTextDatum(nodeToString(varDefexpr)); + else + nulls[Anum_pg_variable_vardefexpr - 1] = true; + + tupdesc = RelationGetDescr(rel); + + varacl = get_user_default_acl(OBJECT_VARIABLE, varOwner, + varNamespace); + + if (varacl != NULL) + values[Anum_pg_variable_varacl - 1] = PointerGetDatum(varacl); + else + nulls[Anum_pg_variable_varacl - 1] = true; + + tup = heap_form_tuple(tupdesc, values, nulls); + CatalogTupleInsert(rel, tup); + Assert(OidIsValid(varid)); + + addrs = new_object_addresses(); + + ObjectAddressSet(myself, VariableRelationId, varid); + + /* dependency on namespace */ + ObjectAddressSet(referenced, NamespaceRelationId, varNamespace); + add_exact_object_address(&referenced, addrs); + + /* dependency on used type */ + ObjectAddressSet(referenced, TypeRelationId, varType); + add_exact_object_address(&referenced, addrs); + + /* dependency on collation */ + if (OidIsValid(varCollation) && + varCollation != DEFAULT_COLLATION_OID) + { + ObjectAddressSet(referenced, CollationRelationId, varCollation); + add_exact_object_address(&referenced, addrs); + } + + /* dependency on default expr */ + if (varDefexpr) + recordDependencyOnExpr(&myself, (Node *) varDefexpr, + NIL, DEPENDENCY_NORMAL); + + record_object_address_dependencies(&myself, addrs, DEPENDENCY_NORMAL); + free_object_addresses(addrs); + + /* dependency on owner */ + recordDependencyOnOwner(VariableRelationId, varid, varOwner); + + /* dependencies on roles mentioned in default ACL */ + recordDependencyOnNewAcl(VariableRelationId, varid, 0, varOwner, varacl); + + /* dependency on extension */ + recordDependencyOnCurrentExtension(&myself, false); + + heap_freetuple(tup); + + /* Post creation hook for new function */ + InvokeObjectPostCreateHook(VariableRelationId, varid, 0); + + table_close(rel, RowExclusiveLock); + + return myself; +} + +/* + * Creates a new variable + * + * Used by CREATE VARIABLE command + */ +ObjectAddress +CreateVariable(ParseState *pstate, CreateSessionVarStmt *stmt) +{ + Oid namespaceid; + AclResult aclresult; + Oid typid; + int32 typmod; + Oid varowner = GetUserId(); + Oid collation; + Oid typcollation; + ObjectAddress variable; + + Node *cooked_default = NULL; + + /* Check consistency of arguments */ + if (stmt->eoxaction == VARIABLE_EOX_DROP + && stmt->variable->relpersistence != RELPERSISTENCE_TEMP) + ereport(ERROR, + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), + errmsg("ON COMMIT DROP can only be used on temporary variables"))); + + if (stmt->is_not_null && stmt->is_immutable && !stmt->defexpr) + ereport(ERROR, + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), + errmsg("IMMUTABLE NOT NULL variable requires default expression"))); + + namespaceid = + RangeVarGetAndCheckCreationNamespace(stmt->variable, NoLock, NULL); + + typenameTypeIdAndMod(pstate, stmt->typeName, &typid, &typmod); + + /* Disallow pseudotypes */ + if (get_typtype(typid) == TYPTYPE_PSEUDO) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("session variable cannot be pseudo-type %s", + format_type_be(typid)))); + + aclresult = object_aclcheck(TypeRelationId, typid, GetUserId(), ACL_USAGE); + if (aclresult != ACLCHECK_OK) + aclcheck_error_type(aclresult, typid); + + typcollation = get_typcollation(typid); + + if (stmt->collClause) + collation = LookupCollation(pstate, + stmt->collClause->collname, + stmt->collClause->location); + else + collation = typcollation;; + + /* Complain if COLLATE is applied to an uncollatable type */ + if (OidIsValid(collation) && !OidIsValid(typcollation)) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("collations are not supported by type %s", + format_type_be(typid)), + parser_errposition(pstate, stmt->collClause->location))); + + if (stmt->defexpr) + { + cooked_default = transformExpr(pstate, stmt->defexpr, + EXPR_KIND_VARIABLE_DEFAULT); + + cooked_default = coerce_to_specific_type(pstate, + cooked_default, typid, "DEFAULT"); + assign_expr_collations(pstate, cooked_default); + } + + variable = create_variable(stmt->variable->relname, + namespaceid, + typid, + typmod, + varowner, + collation, + cooked_default, + stmt->eoxaction, + stmt->is_not_null, + stmt->if_not_exists, + stmt->is_immutable); + + elog(DEBUG1, "record for session variable \"%s\" (oid:%d) was created in pg_variable", + stmt->variable->relname, variable.objectId); + + /* We want SessionVariableCreatePostprocess to see the catalog changes. */ + CommandCounterIncrement(); + + SessionVariableCreatePostprocess(variable.objectId, stmt->eoxaction); + + return variable; +} + +/* + * Drop variable by OID, and register the needed session variable + * cleanup. + */ +void +DropVariable(Oid varid) +{ + Relation rel; + HeapTuple tup; + + rel = table_open(VariableRelationId, RowExclusiveLock); + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for variable %u", varid); + + CatalogTupleDelete(rel, &tup->t_self); + + ReleaseSysCache(tup); + + table_close(rel, RowExclusiveLock); + + /* Do the necessary cleanup if needed in local memory */ + SessionVariableDropPostprocess(varid); +} + +/* + * Fetch attributes (without acl) of session variable from the syscache. + * We don't work with acl directly, so we don't need to read it here. + * Skip deserialization of defexpr when fast_only is true. + */ +void +InitVariable(Variable *var, Oid varid, bool fast_only) +{ + HeapTuple tup; + Form_pg_variable varform; + Datum defexpr_datum; + bool defexpr_isnull; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for session variable %u", varid); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + var->oid = varid; + var->create_lsn = varform->varcreate_lsn; + var->name = pstrdup(NameStr(varform->varname)); + var->namespaceid = varform->varnamespace; + var->typid = varform->vartype; + var->typmod = varform->vartypmod; + var->owner = varform->varowner; + var->collation = varform->varcollation; + var->is_immutable = varform->varisimmutable; + var->is_not_null = varform->varisnotnull; + var->eoxaction = varform->vareoxaction; + + /* Get defexpr */ + defexpr_datum = SysCacheGetAttr(VARIABLEOID, + tup, + Anum_pg_variable_vardefexpr, + &defexpr_isnull); + + var->has_defexpr = !defexpr_isnull; + + /* + * Deserialize defexpr only when it is requested. We need to deserialize + * Node with default expression, only when we read from session variable, + * and this session variable has not assigned value, and this session + * variable has default expression. For other cases, we skip skip this + * operation. + */ + if (!fast_only && !defexpr_isnull) + var->defexpr = stringToNode(TextDatumGetCString(defexpr_datum)); + else + var->defexpr = NULL; + + ReleaseSysCache(tup); +} diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile index 48f7348f91..1cfaeca51e 100644 --- a/src/backend/commands/Makefile +++ b/src/backend/commands/Makefile @@ -50,6 +50,7 @@ OBJS = \ schemacmds.o \ seclabel.o \ sequence.o \ + session_variable.o \ statscmds.o \ subscriptioncmds.o \ tablecmds.o \ diff --git a/src/backend/commands/alter.c b/src/backend/commands/alter.c index 10f28f94bc..9ca83beb1c 100644 --- a/src/backend/commands/alter.c +++ b/src/backend/commands/alter.c @@ -41,6 +41,7 @@ #include "catalog/pg_ts_dict.h" #include "catalog/pg_ts_parser.h" #include "catalog/pg_ts_template.h" +#include "catalog/pg_variable.h" #include "commands/alter.h" #include "commands/collationcmds.h" #include "commands/conversioncmds.h" @@ -143,6 +144,10 @@ report_namespace_conflict(Oid classId, const char *name, Oid nspOid) Assert(OidIsValid(nspOid)); msgfmt = gettext_noop("text search configuration \"%s\" already exists in schema \"%s\""); break; + case VariableRelationId: + Assert(OidIsValid(nspOid)); + msgfmt = gettext_noop("session variable \"%s\" already exists in schema \"%s\""); + break; default: elog(ERROR, "unsupported object class: %u", classId); break; @@ -421,6 +426,7 @@ ExecRenameStmt(RenameStmt *stmt) case OBJECT_TSTEMPLATE: case OBJECT_PUBLICATION: case OBJECT_SUBSCRIPTION: + case OBJECT_VARIABLE: { ObjectAddress address; Relation catalog; @@ -564,6 +570,7 @@ ExecAlterObjectSchemaStmt(AlterObjectSchemaStmt *stmt, case OBJECT_TSDICTIONARY: case OBJECT_TSPARSER: case OBJECT_TSTEMPLATE: + case OBJECT_VARIABLE: { Relation catalog; Relation relation; @@ -654,6 +661,7 @@ AlterObjectNamespace_oid(Oid classId, Oid objid, Oid nspOid, case OCLASS_TSDICT: case OCLASS_TSTEMPLATE: case OCLASS_TSCONFIG: + case OCLASS_VARIABLE: { Relation catalog; @@ -914,6 +922,7 @@ ExecAlterOwnerStmt(AlterOwnerStmt *stmt) case OBJECT_TABLESPACE: case OBJECT_TSDICTIONARY: case OBJECT_TSCONFIGURATION: + case OBJECT_VARIABLE: { Relation catalog; Relation relation; diff --git a/src/backend/commands/dropcmds.c b/src/backend/commands/dropcmds.c index 82bda15889..a5423b687e 100644 --- a/src/backend/commands/dropcmds.c +++ b/src/backend/commands/dropcmds.c @@ -481,6 +481,10 @@ does_not_exist_skipping(ObjectType objtype, Node *object) msg = gettext_noop("publication \"%s\" does not exist, skipping"); name = strVal(object); break; + case OBJECT_VARIABLE: + msg = gettext_noop("session variable \"%s\" does not exist, skipping"); + name = NameListToString(castNode(List, object)); + break; case OBJECT_COLUMN: case OBJECT_DATABASE: diff --git a/src/backend/commands/event_trigger.c b/src/backend/commands/event_trigger.c index d4b00d1a82..ed2a9d776b 100644 --- a/src/backend/commands/event_trigger.c +++ b/src/backend/commands/event_trigger.c @@ -991,6 +991,7 @@ EventTriggerSupportsObjectType(ObjectType obtype) case OBJECT_TSTEMPLATE: case OBJECT_TYPE: case OBJECT_USER_MAPPING: + case OBJECT_VARIABLE: case OBJECT_VIEW: return true; @@ -1057,6 +1058,7 @@ EventTriggerSupportsObjectClass(ObjectClass objclass) case OCLASS_PUBLICATION_REL: case OCLASS_SUBSCRIPTION: case OCLASS_TRANSFORM: + case OCLASS_VARIABLE: return true; /* @@ -2049,6 +2051,8 @@ stringify_grant_objtype(ObjectType objtype) return "TABLESPACE"; case OBJECT_TYPE: return "TYPE"; + case OBJECT_VARIABLE: + return "VARIABLE"; /* these currently aren't used */ case OBJECT_ACCESS_METHOD: case OBJECT_AGGREGATE: @@ -2132,6 +2136,8 @@ stringify_adefprivs_objtype(ObjectType objtype) return "TABLESPACES"; case OBJECT_TYPE: return "TYPES"; + case OBJECT_VARIABLE: + return "VARIABLES"; /* these currently aren't used */ case OBJECT_ACCESS_METHOD: case OBJECT_AGGREGATE: diff --git a/src/backend/commands/seclabel.c b/src/backend/commands/seclabel.c index 7ff16e3276..75b39b2945 100644 --- a/src/backend/commands/seclabel.c +++ b/src/backend/commands/seclabel.c @@ -92,6 +92,7 @@ SecLabelSupportsObjectType(ObjectType objtype) case OBJECT_TSPARSER: case OBJECT_TSTEMPLATE: case OBJECT_USER_MAPPING: + case OBJECT_VARIABLE: return false; /* diff --git a/src/backend/commands/session_variable.c b/src/backend/commands/session_variable.c new file mode 100644 index 0000000000..946e73e467 --- /dev/null +++ b/src/backend/commands/session_variable.c @@ -0,0 +1,243 @@ +/*------------------------------------------------------------------------- + * + * session_variable.c + * session variable creation/manipulation commands + * + * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/commands/sessionvariable.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/heapam.h" +#include "catalog/dependency.h" +#include "catalog/indexing.h" +#include "catalog/namespace.h" +#include "catalog/pg_variable.h" +#include "commands/session_variable.h" +#include "funcapi.h" +#include "miscadmin.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/memutils.h" +#include "utils/syscache.h" + +/* + * The life cycle of temporary session variable can be + * limmited by using clause ON COMMIT DROP. + */ +typedef enum SVariableXActAction +{ + SVAR_ON_COMMIT_DROP, /* used for ON COMMIT DROP */ +} SVariableXActAction; + +typedef struct SVariableXActActionItem +{ + Oid varid; /* varid of session variable */ + + /* + * creating_subid is the ID of the creating subxact. If the action was + * unregistered during the current transaction, deleting_subid is the ID + * of the deleting subxact, otherwise InvalidSubTransactionId. + */ + SubTransactionId creating_subid; + SubTransactionId deleting_subid; +} SVariableXActActionItem; + +/* List holds fields of SVariableXActActionItem type */ +static List *xact_drop_actions = NIL; + +static void register_session_variable_xact_action(Oid varid, SVariableXActAction action); +static void unregister_session_variable_xact_action(Oid varid, SVariableXActAction action); + + +/* + * Do the necessary work to setup local memory management of a new + * variable. + * + * Caller should already have created the necessary entry in catalog + * and made them visible. + */ +void +SessionVariableCreatePostprocess(Oid varid, char eoxaction) +{ + /* + * For temporary variables, we need to create a new end of xact action to + * ensure deletion from catalog. + */ + if (eoxaction == VARIABLE_EOX_DROP) + { + Assert(isTempNamespace(get_session_variable_namespace(varid))); + + register_session_variable_xact_action(varid, SVAR_ON_COMMIT_DROP); + } +} + +/* + * Handle the local memory cleanup for a DROP VARIABLE command. + * + * Caller should take care of removing the pg_variable entry first. + */ +void +SessionVariableDropPostprocess(Oid varid) +{ + /* + * The entry was removed from catalog already, we must not do it + * again at end of xact time. + */ + unregister_session_variable_xact_action(varid, SVAR_ON_COMMIT_DROP); +} + +/* + * Registration of actions to be executed on session variables at transaction + * end time. We want to drop temporary session variables with clause ON COMMIT + * DROP, or we want to reset values of session variables with clause ON + * TRANSACTION END RESET or we want to clean (reset) local memory allocated by + * values of dropped session variables. + */ + +/* + * Register a session variable xact action. + */ +static void +register_session_variable_xact_action(Oid varid, + SVariableXActAction action) +{ + SVariableXActActionItem *xact_ai; + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(CacheMemoryContext); + + xact_ai = (SVariableXActActionItem *) + palloc(sizeof(SVariableXActActionItem)); + + xact_ai->varid = varid; + + xact_ai->creating_subid = GetCurrentSubTransactionId(); + xact_ai->deleting_subid = InvalidSubTransactionId; + + Assert(action == SVAR_ON_COMMIT_DROP); + xact_drop_actions = lcons(xact_ai, xact_drop_actions); + + MemoryContextSwitchTo(oldcxt); +} + +/* + * Unregister an action on a given session variable from action list. In this + * moment, the action is just marked as deleted by setting deleting_subid. The + * calling even might be rollbacked, in which case we should not lose this + * action. + */ +static void +unregister_session_variable_xact_action(Oid varid, + SVariableXActAction action) +{ + ListCell *l; + + Assert(action == SVAR_ON_COMMIT_DROP); + + foreach(l, xact_drop_actions) + { + SVariableXActActionItem *xact_ai = + (SVariableXActActionItem *) lfirst(l); + + if (xact_ai->varid == varid) + xact_ai->deleting_subid = GetCurrentSubTransactionId(); + } +} + +/* + * Perform ON TRANSACTION END RESET or ON COMMIT DROP + * and COMMIT/ROLLBACK of transaction session variables. + */ +void +AtPreEOXact_SessionVariable(bool isCommit) +{ + ListCell *l; + + foreach(l, xact_drop_actions) + { + SVariableXActActionItem *xact_ai = + (SVariableXActActionItem *) lfirst(l); + + /* Iterate only over entries that are still pending */ + if (xact_ai->deleting_subid == InvalidSubTransactionId) + { + + /* + * ON COMMIT DROP is allowed only for temp session variables. So + * we should explicitly delete only when current transaction was + * committed. When it's rollback, then session variable is removed + * automatically. + */ + if (isCommit) + { + ObjectAddress object; + + object.classId = VariableRelationId; + object.objectId = xact_ai->varid; + object.objectSubId = 0; + + /* + * Since this is an automatic drop, rather than one directly + * initiated by the user, we pass the + * PERFORM_DELETION_INTERNAL flag. + */ + elog(DEBUG1, "session variable (oid:%u) will be deleted (forced by SVAR_ON_COMMIT_DROP action)", + xact_ai->varid); + + performDeletion(&object, DROP_CASCADE, + PERFORM_DELETION_INTERNAL | + PERFORM_DELETION_QUIETLY); + } + } + } + + /* + * Any drop action left is an entry that was unregistered and not + * rollbacked, so we can simply remove them. + */ + list_free_deep(xact_drop_actions); + xact_drop_actions = NIL; +} + +/* + * Post-subcommit or post-subabort cleanup of xact action list. + * + * During subabort, we can immediately remove entries created during this + * subtransaction. During subcommit, just transfer entries marked during + * this subtransaction as being the parent's responsibility. + */ +void +AtEOSubXact_SessionVariable(bool isCommit, + SubTransactionId mySubid, + SubTransactionId parentSubid) +{ + ListCell *cur_item; + + foreach(cur_item, xact_drop_actions) + { + SVariableXActActionItem *xact_ai = + (SVariableXActActionItem *) lfirst(cur_item); + + if (!isCommit && xact_ai->creating_subid == mySubid) + { + /* cur_item must be removed */ + xact_drop_actions = foreach_delete_current(xact_drop_actions, cur_item); + pfree(xact_ai); + } + else + { + /* cur_item must be preserved */ + if (xact_ai->creating_subid == mySubid) + xact_ai->creating_subid = parentSubid; + if (xact_ai->deleting_subid == mySubid) + xact_ai->deleting_subid = isCommit ? parentSubid : InvalidSubTransactionId; + } + } +} diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index c7a8a689b7..c99f091f1f 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -47,6 +47,7 @@ #include "catalog/pg_tablespace.h" #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "catalog/storage.h" #include "catalog/storage_xlog.h" #include "catalog/toasting.h" @@ -6487,6 +6488,8 @@ ATTypedTableRecursion(List **wqueue, Relation rel, AlterTableCmd *cmd, * Eventually, we'd like to propagate the check or rewrite operation * into such tables, but for now, just error out if we find any. * + * Check if the type "typeOid" is used as type of some session variable too. + * * Caller should provide either the associated relation of a rowtype, * or a type name (not both) for use in the error message, if any. * @@ -6549,6 +6552,45 @@ find_composite_type_dependencies(Oid typeOid, Relation origRelation, continue; } + /* Don't allow change of type used by session's variable */ + if (pg_depend->classid == VariableRelationId) + { + Oid varid = pg_depend->objid; + + if (origTypeName) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot alter type \"%s\" because session variable \"%s.%s\" uses it", + origTypeName, + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)))); + else if (origRelation->rd_rel->relkind == RELKIND_COMPOSITE_TYPE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot alter type \"%s\" because session variable \"%s.%s\" uses it", + RelationGetRelationName(origRelation), + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)))); + else if (origRelation->rd_rel->relkind == RELKIND_FOREIGN_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot alter foreign table \"%s\" because session variable \"%s.%s\" uses it", + RelationGetRelationName(origRelation), + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)))); + else if (origRelation->rd_rel->relkind == RELKIND_RELATION || + origRelation->rd_rel->relkind == RELKIND_MATVIEW || + origRelation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot alter table \"%s\" because session variable \"%s.%s\" uses it", + RelationGetRelationName(origRelation), + get_namespace_name(get_session_variable_namespace(varid)), + get_session_variable_name(varid)))); + + continue; + } + /* Else, ignore dependees that aren't relations */ if (pg_depend->classid != RelationRelationId) continue; @@ -12835,6 +12877,7 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, case OCLASS_PUBLICATION_REL: case OCLASS_SUBSCRIPTION: case OCLASS_TRANSFORM: + case OCLASS_VARIABLE: /* * We don't expect any of these sorts of objects to depend on diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 6f5aa8a3cb..f5533d2472 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -53,6 +53,7 @@ #include "catalog/namespace.h" #include "catalog/pg_am.h" #include "catalog/pg_trigger.h" +#include "catalog/pg_variable.h" #include "commands/defrem.h" #include "commands/trigger.h" #include "gramparse.h" @@ -294,8 +295,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt - CreateSchemaStmt CreateSeqStmt CreateStmt CreateStatsStmt CreateTableSpaceStmt - CreateFdwStmt CreateForeignServerStmt CreateForeignTableStmt + CreateSchemaStmt CreateSessionVarStmt CreateSeqStmt CreateStmt CreateStatsStmt + CreateTableSpaceStmt CreateFdwStmt CreateForeignServerStmt CreateForeignTableStmt CreateAssertionStmt CreateTransformStmt CreateTrigStmt CreateEventTrigStmt CreateUserStmt CreateUserMappingStmt CreateRoleStmt CreatePolicyStmt CreatedbStmt DeclareCursorStmt DefineStmt DeleteStmt DiscardStmt DoStmt @@ -475,6 +476,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <ival> OptTemp %type <ival> OptNoLog %type <oncommit> OnCommitOption +%type <ival> OnEOXActionOption %type <ival> for_locking_strength %type <node> for_locking_item @@ -644,6 +646,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <partboundspec> PartitionBoundSpec %type <list> hash_partbound %type <defelt> hash_partbound_elem +%type <node> OptSessionVarDefExpr +%type <boolean> OptNotNull OptImmutable %type <node> json_format_clause_opt @@ -772,8 +776,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); UESCAPE UNBOUNDED UNCOMMITTED UNENCRYPTED UNION UNIQUE UNKNOWN UNLISTEN UNLOGGED UNTIL UPDATE USER USING - VACUUM VALID VALIDATE VALIDATOR VALUE_P VALUES VARCHAR VARIADIC VARYING - VERBOSE VERSION_P VIEW VIEWS VOLATILE + VACUUM VALID VALIDATE VALIDATOR VALUE_P VALUES VARCHAR VARIABLE VARIABLES + VARIADIC VARYING VERBOSE VERSION_P VIEW VIEWS VOLATILE WHEN WHERE WHITESPACE_P WINDOW WITH WITHIN WITHOUT WORK WRAPPER WRITE @@ -1027,6 +1031,7 @@ stmt: | CreatePolicyStmt | CreatePLangStmt | CreateSchemaStmt + | CreateSessionVarStmt | CreateSeqStmt | CreateStmt | CreateSubscriptionStmt @@ -1571,6 +1576,7 @@ schema_stmt: | CreateTrigStmt | GrantStmt | ViewStmt + | CreateSessionVarStmt ; @@ -5056,6 +5062,69 @@ create_extension_opt_item: } ; +/***************************************************************************** + * + * QUERY : + * CREATE VARIABLE varname [AS] type + * + *****************************************************************************/ + +CreateSessionVarStmt: + CREATE OptTemp OptImmutable VARIABLE qualified_name opt_as Typename opt_collate_clause OptNotNull OptSessionVarDefExpr OnEOXActionOption + { + CreateSessionVarStmt *n = makeNode(CreateSessionVarStmt); + $5->relpersistence = $2; + n->is_immutable = $3; + n->variable = $5; + n->typeName = $7; + n->collClause = (CollateClause *) $8; + n->is_not_null = $9; + n->defexpr = $10; + n->eoxaction = $11; + n->if_not_exists = false; + $$ = (Node *) n; + } + | CREATE OptTemp OptImmutable VARIABLE IF_P NOT EXISTS qualified_name opt_as Typename opt_collate_clause OptNotNull OptSessionVarDefExpr OnEOXActionOption + { + CreateSessionVarStmt *n = makeNode(CreateSessionVarStmt); + $8->relpersistence = $2; + n->is_immutable = $3; + n->variable = $8; + n->typeName = $10; + n->collClause = (CollateClause *) $11; + n->is_not_null = $12; + n->defexpr = $13; + n->eoxaction = $14; + n->if_not_exists = true; + $$ = (Node *) n; + } + ; + +OptSessionVarDefExpr: DEFAULT b_expr { $$ = $2; } + | /* EMPTY */ { $$ = NULL; } + ; + +/* + * Temporary session variables can be dropped on successful + * transaction end like tables. RESET can only be forced on + * transaction end. Since the session variables are not + * transactional, we have to handle ROLLBACK too. + * The clause ON TRANSACTION END is clearer than some + * ON COMMIT ROLLBACK RESET clause. + */ +OnEOXActionOption: ON COMMIT DROP { $$ = VARIABLE_EOX_DROP; } + | ON TRANSACTION END_P RESET { $$ = VARIABLE_EOX_RESET; } + | /*EMPTY*/ { $$ = VARIABLE_EOX_NOOP; } + ; + +OptNotNull: NOT NULL_P { $$ = true; } + | /* EMPTY */ { $$ = false; } + ; + +OptImmutable: IMMUTABLE { $$ = true; } + | /* EMPTY */ { $$ = false; } + ; + /***************************************************************************** * * ALTER EXTENSION name UPDATE [ TO version ] @@ -6833,6 +6902,7 @@ object_type_any_name: | TEXT_P SEARCH DICTIONARY { $$ = OBJECT_TSDICTIONARY; } | TEXT_P SEARCH TEMPLATE { $$ = OBJECT_TSTEMPLATE; } | TEXT_P SEARCH CONFIGURATION { $$ = OBJECT_TSCONFIGURATION; } + | VARIABLE { $$ = OBJECT_VARIABLE; } ; /* @@ -7709,6 +7779,14 @@ privilege_target: n->objs = $2; $$ = n; } + | VARIABLE qualified_name_list + { + PrivTarget *n = (PrivTarget *) palloc(sizeof(PrivTarget)); + n->targtype = ACL_TARGET_OBJECT; + n->objtype = OBJECT_VARIABLE; + n->objs = $2; + $$ = n; + } | ALL TABLES IN_P SCHEMA name_list { PrivTarget *n = (PrivTarget *) palloc(sizeof(PrivTarget)); @@ -7754,6 +7832,14 @@ privilege_target: n->objs = $5; $$ = n; } + | ALL VARIABLES IN_P SCHEMA name_list + { + PrivTarget *n = (PrivTarget *) palloc(sizeof(PrivTarget)); + n->targtype = ACL_TARGET_ALL_IN_SCHEMA; + n->objtype = OBJECT_VARIABLE; + n->objs = $5; + $$ = n; + } ; @@ -7951,6 +8037,7 @@ defacl_privilege_target: | SEQUENCES { $$ = OBJECT_SEQUENCE; } | TYPES_P { $$ = OBJECT_TYPE; } | SCHEMAS { $$ = OBJECT_SCHEMA; } + | VARIABLES { $$ = OBJECT_VARIABLE; } ; @@ -9733,6 +9820,25 @@ RenameStmt: ALTER AGGREGATE aggregate_with_argtypes RENAME TO name n->missing_ok = false; $$ = (Node *) n; } + | ALTER VARIABLE any_name RENAME TO name + { + RenameStmt *n = makeNode(RenameStmt); + n->renameType = OBJECT_VARIABLE; + n->object = (Node *) $3; + n->newname = $6; + n->missing_ok = false; + $$ = (Node *)n; + } + | ALTER VARIABLE IF_P EXISTS any_name RENAME TO name + { + RenameStmt *n = makeNode(RenameStmt); + n->renameType = OBJECT_VARIABLE; + n->object = (Node *) $5; + n->newname = $8; + n->missing_ok = true; + $$ = (Node *)n; + } + ; opt_column: COLUMN @@ -10094,6 +10200,25 @@ AlterObjectSchemaStmt: n->missing_ok = false; $$ = (Node *) n; } + | ALTER VARIABLE any_name SET SCHEMA name + { + AlterObjectSchemaStmt *n = makeNode(AlterObjectSchemaStmt); + n->objectType = OBJECT_VARIABLE; + n->object = (Node *) $3; + n->newschema = $6; + n->missing_ok = false; + $$ = (Node *)n; + } + | ALTER VARIABLE IF_P EXISTS any_name SET SCHEMA name + { + AlterObjectSchemaStmt *n = makeNode(AlterObjectSchemaStmt); + n->objectType = OBJECT_VARIABLE; + n->object = (Node *) $5; + n->newschema = $8; + n->missing_ok = true; + $$ = (Node *)n; + } + ; /***************************************************************************** @@ -10373,6 +10498,14 @@ AlterOwnerStmt: ALTER AGGREGATE aggregate_with_argtypes OWNER TO RoleSpec n->newowner = $6; $$ = (Node *) n; } + | ALTER VARIABLE any_name OWNER TO RoleSpec + { + AlterOwnerStmt *n = makeNode(AlterOwnerStmt); + n->objectType = OBJECT_VARIABLE; + n->object = (Node *) $3; + n->newowner = $6; + $$ = (Node *)n; + } ; @@ -17256,6 +17389,8 @@ unreserved_keyword: | VALIDATE | VALIDATOR | VALUE_P + | VARIABLE + | VARIABLES | VARYING | VERSION_P | VIEW @@ -17881,6 +18016,8 @@ bare_label_keyword: | VALUE_P | VALUES | VARCHAR + | VARIABLE + | VARIABLES | VARIADIC | VERBOSE | VERSION_P diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..6fc03166c6 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -472,6 +472,7 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) break; case EXPR_KIND_COLUMN_DEFAULT: case EXPR_KIND_FUNCTION_DEFAULT: + case EXPR_KIND_VARIABLE_DEFAULT: if (isAgg) err = _("aggregate functions are not allowed in DEFAULT expressions"); @@ -915,6 +916,7 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, break; case EXPR_KIND_COLUMN_DEFAULT: case EXPR_KIND_FUNCTION_DEFAULT: + case EXPR_KIND_VARIABLE_DEFAULT: err = _("window functions are not allowed in DEFAULT expressions"); break; case EXPR_KIND_INDEX_EXPRESSION: diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 0b3632735b..327c5956b2 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -18,6 +18,7 @@ #include "catalog/pg_aggregate.h" #include "catalog/pg_proc.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "commands/dbcommands.h" #include "miscadmin.h" #include "nodes/makefuncs.h" @@ -541,6 +542,8 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_VARIABLE_DEFAULT: + /* okay */ break; @@ -1762,6 +1765,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) break; case EXPR_KIND_COLUMN_DEFAULT: case EXPR_KIND_FUNCTION_DEFAULT: + case EXPR_KIND_VARIABLE_DEFAULT: err = _("cannot use subquery in DEFAULT expression"); break; case EXPR_KIND_INDEX_EXPRESSION: @@ -3108,6 +3112,7 @@ ParseExprKindName(ParseExprKind exprKind) return "CHECK"; case EXPR_KIND_COLUMN_DEFAULT: case EXPR_KIND_FUNCTION_DEFAULT: + case EXPR_KIND_VARIABLE_DEFAULT: return "DEFAULT"; case EXPR_KIND_INDEX_EXPRESSION: return "index expression"; diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..a0fdaa8605 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2618,6 +2618,7 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) break; case EXPR_KIND_COLUMN_DEFAULT: case EXPR_KIND_FUNCTION_DEFAULT: + case EXPR_KIND_VARIABLE_DEFAULT: err = _("set-returning functions are not allowed in DEFAULT expressions"); break; case EXPR_KIND_INDEX_EXPRESSION: diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index b1255e3b70..e7e75209dd 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -105,6 +105,7 @@ typedef struct List *indexes; /* CREATE INDEX items */ List *triggers; /* CREATE TRIGGER items */ List *grants; /* GRANT items */ + List *variables; /* CREATE VARIABLE items */ } CreateSchemaStmtContext; @@ -3822,6 +3823,7 @@ transformCreateSchemaStmtElements(List *schemaElts, const char *schemaName) cxt.indexes = NIL; cxt.triggers = NIL; cxt.grants = NIL; + cxt.variables = NIL; /* * Run through each schema element in the schema element list. Separate @@ -3890,6 +3892,15 @@ transformCreateSchemaStmtElements(List *schemaElts, const char *schemaName) cxt.grants = lappend(cxt.grants, element); break; + case T_CreateSessionVarStmt: + { + CreateSessionVarStmt *elp = (CreateSessionVarStmt *) element; + + setSchemaName(cxt.schemaname, &elp->variable->schemaname); + cxt.variables = lappend(cxt.variables, element); + } + break; + default: elog(ERROR, "unrecognized node type: %d", (int) nodeTag(element)); @@ -3903,6 +3914,7 @@ transformCreateSchemaStmtElements(List *schemaElts, const char *schemaName) result = list_concat(result, cxt.indexes); result = list_concat(result, cxt.triggers); result = list_concat(result, cxt.grants); + result = list_concat(result, cxt.variables); return result; } diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 30b51bf4d3..ddc26794b2 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -49,6 +49,7 @@ #include "commands/proclang.h" #include "commands/publicationcmds.h" #include "commands/schemacmds.h" +#include "commands/session_variable.h" #include "commands/seclabel.h" #include "commands/sequence.h" #include "commands/subscriptioncmds.h" @@ -189,6 +190,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree) case T_CreateRangeStmt: case T_CreateRoleStmt: case T_CreateSchemaStmt: + case T_CreateSessionVarStmt: case T_CreateSeqStmt: case T_CreateStatsStmt: case T_CreateStmt: @@ -1396,6 +1398,10 @@ ProcessUtilitySlow(ParseState *pstate, } break; + case T_CreateSessionVarStmt: + address = CreateVariable(pstate, (CreateSessionVarStmt *) parsetree); + break; + /* * ************* object creation / destruction ************** */ @@ -2342,6 +2348,9 @@ AlterObjectTypeCommandTag(ObjectType objtype) case OBJECT_STATISTIC_EXT: tag = CMDTAG_ALTER_STATISTICS; break; + case OBJECT_VARIABLE: + tag = CMDTAG_ALTER_VARIABLE; + break; default: tag = CMDTAG_UNKNOWN; break; @@ -2650,6 +2659,9 @@ CreateCommandTag(Node *parsetree) case OBJECT_STATISTIC_EXT: tag = CMDTAG_DROP_STATISTICS; break; + case OBJECT_VARIABLE: + tag = CMDTAG_DROP_VARIABLE; + break; default: tag = CMDTAG_UNKNOWN; } @@ -3226,6 +3238,10 @@ CreateCommandTag(Node *parsetree) } break; + case T_CreateSessionVarStmt: + tag = CMDTAG_CREATE_VARIABLE; + break; + default: elog(WARNING, "unrecognized node type: %d", (int) nodeTag(parsetree)); diff --git a/src/backend/utils/adt/acl.c b/src/backend/utils/adt/acl.c index c660fd3e70..dc09de12b9 100644 --- a/src/backend/utils/adt/acl.c +++ b/src/backend/utils/adt/acl.c @@ -844,6 +844,10 @@ acldefault(ObjectType objtype, Oid ownerId) world_default = ACL_NO_RIGHTS; owner_default = ACL_ALL_RIGHTS_PARAMETER_ACL; break; + case OBJECT_VARIABLE: + world_default = ACL_NO_RIGHTS; + owner_default = ACL_ALL_RIGHTS_VARIABLE; + break; default: elog(ERROR, "unrecognized object type: %d", (int) objtype); world_default = ACL_NO_RIGHTS; /* keep compiler quiet */ @@ -941,6 +945,9 @@ acldefault_sql(PG_FUNCTION_ARGS) case 'T': objtype = OBJECT_TYPE; break; + case 'V': + objtype = OBJECT_VARIABLE; + break; default: elog(ERROR, "unrecognized object type abbreviation: %c", objtypec); } diff --git a/src/backend/utils/cache/lsyscache.c b/src/backend/utils/cache/lsyscache.c index c7607895cd..703e261de7 100644 --- a/src/backend/utils/cache/lsyscache.c +++ b/src/backend/utils/cache/lsyscache.c @@ -36,6 +36,7 @@ #include "catalog/pg_subscription.h" #include "catalog/pg_transform.h" #include "catalog/pg_type.h" +#include "catalog/pg_variable.h" #include "miscadmin.h" #include "nodes/makefuncs.h" #include "utils/array.h" @@ -3672,3 +3673,115 @@ get_subscription_name(Oid subid, bool missing_ok) return subname; } + +/* ---------- PG_VARIABLE CACHE ---------- */ + +/* + * get_varname_varid + * Given name and namespace of variable, look up the OID. + */ +Oid +get_varname_varid(const char *varname, Oid varnamespace) +{ + return GetSysCacheOid2(VARIABLENAMENSP, Anum_pg_variable_oid, + PointerGetDatum(varname), + ObjectIdGetDatum(varnamespace)); +} + +/* + * get_session_variable_name + * Returns a palloc'd copy of the name of a given session variable. + */ +char * +get_session_variable_name(Oid varid) +{ + HeapTuple tup; + Form_pg_variable varform; + char *varname; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for session variable %u", varid); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + varname = pstrdup(NameStr(varform->varname)); + + ReleaseSysCache(tup); + + return varname; +} + +/* + * get_session_variable_namespace + * Returns the pg_namespace OID associated with a given session variable. + */ +Oid +get_session_variable_namespace(Oid varid) +{ + HeapTuple tup; + Form_pg_variable varform; + Oid varnamespace; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for variable %u", varid); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + varnamespace = varform->varnamespace; + + ReleaseSysCache(tup); + + return varnamespace; +} + +/* + * Returns the type, typmod and collid of the given session variable. + */ +Oid +get_session_variable_type(Oid varid) +{ + HeapTuple tup; + Form_pg_variable varform; + Oid vartype; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for session variable %u", varid); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + vartype = varform->vartype; + + ReleaseSysCache(tup); + + return vartype; +} + +/* + * Returns the type, typmod and collid of the given session variable. + */ +void +get_session_variable_type_typmod_collid(Oid varid, Oid *typid, int32 *typmod, + Oid *collid) +{ + HeapTuple tup; + Form_pg_variable varform; + + tup = SearchSysCache1(VARIABLEOID, ObjectIdGetDatum(varid)); + + if (!HeapTupleIsValid(tup)) + elog(ERROR, "cache lookup failed for session variable %u", varid); + + varform = (Form_pg_variable) GETSTRUCT(tup); + + *typid = varform->vartype; + *typmod = varform->vartypmod; + *collid = varform->varcollation; + + ReleaseSysCache(tup); +} diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c index 4e4a34bde8..d0e5265519 100644 --- a/src/backend/utils/cache/syscache.c +++ b/src/backend/utils/cache/syscache.c @@ -75,6 +75,7 @@ #include "catalog/pg_ts_template.h" #include "catalog/pg_type.h" #include "catalog/pg_user_mapping.h" +#include "catalog/pg_variable.h" #include "lib/qunique.h" #include "utils/catcache.h" #include "utils/lsyscache.h" @@ -676,6 +677,19 @@ static const struct cachedesc cacheinfo[] = { KEY(Anum_pg_user_mapping_umuser, Anum_pg_user_mapping_umserver), 2 + }, + [VARIABLENAMENSP] = { + VariableRelationId, + VariableNameNspIndexId, + KEY(Anum_pg_variable_varname, + Anum_pg_variable_varnamespace), + 8 + }, + [VARIABLEOID] = { + VariableRelationId, + VariableObjectIndexId, + KEY(Anum_pg_variable_oid), + 8 } }; diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h index ffd5e9dc82..e43a1ab96a 100644 --- a/src/include/catalog/dependency.h +++ b/src/include/catalog/dependency.h @@ -127,10 +127,11 @@ typedef enum ObjectClass OCLASS_PUBLICATION_NAMESPACE, /* pg_publication_namespace */ OCLASS_PUBLICATION_REL, /* pg_publication_rel */ OCLASS_SUBSCRIPTION, /* pg_subscription */ - OCLASS_TRANSFORM /* pg_transform */ + OCLASS_TRANSFORM, /* pg_transform */ + OCLASS_VARIABLE /* pg_variable */ } ObjectClass; -#define LAST_OCLASS OCLASS_TRANSFORM +#define LAST_OCLASS OCLASS_VARIABLE /* flag bits for performDeletion/performMultipleDeletions: */ #define PERFORM_DELETION_INTERNAL 0x0001 /* internal action */ diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build index 3179be09d3..51414b93bd 100644 --- a/src/include/catalog/meson.build +++ b/src/include/catalog/meson.build @@ -65,6 +65,7 @@ catalog_headers = [ 'pg_publication_rel.h', 'pg_subscription.h', 'pg_subscription_rel.h', + 'pg_variable.h', ] bki_data = [ diff --git a/src/include/catalog/namespace.h b/src/include/catalog/namespace.h index f64a0ec26b..787de15ed1 100644 --- a/src/include/catalog/namespace.h +++ b/src/include/catalog/namespace.h @@ -96,6 +96,8 @@ extern Oid TypenameGetTypid(const char *typname); extern Oid TypenameGetTypidExtended(const char *typname, bool temp_ok); extern bool TypeIsVisible(Oid typid); +extern bool VariableIsVisible(Oid varid); + extern FuncCandidateList FuncnameGetCandidates(List *names, int nargs, List *argnames, bool expand_variadic, @@ -164,6 +166,9 @@ extern void SetTempNamespaceState(Oid tempNamespaceId, Oid tempToastNamespaceId); extern void ResetTempTableNamespace(void); +extern List *NamesFromList(List *names); +extern Oid LookupVariable(const char *nspname, const char *varname, bool missing_ok); + extern OverrideSearchPath *GetOverrideSearchPath(MemoryContext context); extern OverrideSearchPath *CopyOverrideSearchPath(OverrideSearchPath *path); extern bool OverrideSearchPathMatchesCurrent(OverrideSearchPath *path); diff --git a/src/include/catalog/pg_default_acl.h b/src/include/catalog/pg_default_acl.h index d6d0a03f0c..53affc969c 100644 --- a/src/include/catalog/pg_default_acl.h +++ b/src/include/catalog/pg_default_acl.h @@ -66,6 +66,7 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_default_acl_oid_index, 828, DefaultAclOidIndexId, o #define DEFACLOBJ_FUNCTION 'f' /* function */ #define DEFACLOBJ_TYPE 'T' /* type */ #define DEFACLOBJ_NAMESPACE 'n' /* namespace */ +#define DEFACLOBJ_VARIABLE 'V' /* variable */ #endif /* EXPOSE_TO_CLIENT_CODE */ diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index d7a60b39b5..e1034fdc49 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6373,6 +6373,9 @@ proname => 'pg_collation_is_visible', procost => '10', provolatile => 's', prorettype => 'bool', proargtypes => 'oid', prosrc => 'pg_collation_is_visible' }, +{ oid => '9221', descr => 'is session variable visible in search path?', + proname => 'pg_variable_is_visible', procost => '10', provolatile => 's', + prorettype => 'bool', proargtypes => 'oid', prosrc => 'pg_variable_is_visible' }, { oid => '2854', descr => 'get OID of current session\'s temp schema, if any', proname => 'pg_my_temp_schema', provolatile => 's', proparallel => 'r', diff --git a/src/include/catalog/pg_variable.h b/src/include/catalog/pg_variable.h new file mode 100644 index 0000000000..698cd73c7e --- /dev/null +++ b/src/include/catalog/pg_variable.h @@ -0,0 +1,130 @@ +/*------------------------------------------------------------------------- + * + * pg_variable.h + * definition of session variables system catalog (pg_variables) + * + * + * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/catalog/pg_variable.h + * + * NOTES + * The Catalog.pm module reads this file and derives schema + * information. + * + *------------------------------------------------------------------------- + */ +#ifndef PG_VARIABLE_H +#define PG_VARIABLE_H + +#include "catalog/genbki.h" +#include "catalog/objectaddress.h" +#include "catalog/pg_variable_d.h" +#include "utils/acl.h" + +/* ---------------- + * pg_variable definition. cpp turns this into + * typedef struct FormData_pg_variable + * + * The column varcreate_lsn of XlogRecPtr type (8-byte) should be on position + * divisible by 8 unconditionally and before varname column of NameData type. + * see sanity_check:check_columns + * ---------------- + */ +CATALOG(pg_variable,9222,VariableRelationId) +{ + Oid oid; /* oid */ + + /* OID of entry in pg_type for variable's type */ + Oid vartype BKI_LOOKUP(pg_type); + + /* Used for identity check [oid, create_lsn] */ + XLogRecPtr varcreate_lsn; + + /* variable name */ + NameData varname; + + /* OID of namespace containing variable class */ + Oid varnamespace BKI_LOOKUP(pg_namespace); + + /* variable owner */ + Oid varowner BKI_LOOKUP(pg_authid); + + /* typmod for variable's type */ + int32 vartypmod; + + /* Don't allow NULL */ + bool varisnotnull; + + /* Don't allow changes */ + bool varisimmutable; + + /* action on transaction end */ + char vareoxaction; + + /* variable collation */ + Oid varcollation BKI_LOOKUP_OPT(pg_collation); + + +#ifdef CATALOG_VARLEN /* variable-length fields start here */ + + /* access permissions */ + aclitem varacl[1] BKI_DEFAULT(_null_); + + /* list of expression trees for variable default (NULL if none) */ + pg_node_tree vardefexpr BKI_DEFAULT(_null_); + +#endif +} FormData_pg_variable; + +typedef enum VariableEOXAction +{ + VARIABLE_EOX_NOOP = 'n', /* NOOP */ + VARIABLE_EOX_DROP = 'd', /* ON COMMIT DROP */ + VARIABLE_EOX_RESET = 'r', /* ON TRANSACTION END RESET */ +} VariableEOXAction; + +/* ---------------- + * Form_pg_variable corresponds to a pointer to a tuple with + * the format of pg_variable relation. + * ---------------- + */ +typedef FormData_pg_variable *Form_pg_variable; + +DECLARE_TOAST(pg_variable, 9223, 9224); + +DECLARE_UNIQUE_INDEX_PKEY(pg_variable_oid_index, 9225, VariableOidIndexId, on pg_variable using btree(oid oid_ops)); +#define VariableObjectIndexId 9225 + +DECLARE_UNIQUE_INDEX(pg_variable_varname_nsp_index, 9226, VariableNameNspIndexId, on pg_variable using btree(varname name_ops, varnamespace oid_ops)); +#define VariableNameNspIndexId 9226 + +/* + * The Variable struct is based on FormData_pg_variable struct. Unlike + * FormData_pg_variable it can hold node of deserialized expression used + * for calculation of default value. + */ +typedef struct Variable +{ + Oid oid; + Oid typid; + XLogRecPtr create_lsn; + char *name; + Oid namespaceid; + Oid owner; + int32 typmod; + bool is_not_null; + bool is_immutable; + VariableEOXAction eoxaction; + Oid collation; + bool has_defexpr; + Node *defexpr; +} Variable; + +extern ObjectAddress CreateVariable(ParseState *pstate, + CreateSessionVarStmt *stmt); +extern void DropVariable(Oid varid); +extern void InitVariable(Variable *var, Oid varid, bool fast_only); + +#endif /* PG_VARIABLE_H */ diff --git a/src/include/commands/session_variable.h b/src/include/commands/session_variable.h new file mode 100644 index 0000000000..343ee070a5 --- /dev/null +++ b/src/include/commands/session_variable.h @@ -0,0 +1,34 @@ +/*------------------------------------------------------------------------- + * + * sessionvariable.h + * prototypes for sessionvariable.c. + * + * + * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/commands/session_variable.h + * + *------------------------------------------------------------------------- + */ + +#ifndef SESSIONVARIABLE_H +#define SESSIONVARIABLE_H + +#include "catalog/objectaddress.h" +#include "catalog/pg_variable.h" +#include "nodes/params.h" +#include "nodes/parsenodes.h" +#include "nodes/plannodes.h" +#include "tcop/cmdtag.h" +#include "utils/queryenvironment.h" + +extern void SessionVariableCreatePostprocess(Oid varid, char eoxaction); +extern void SessionVariableDropPostprocess(Oid varid); + +extern void AtPreEOXact_SessionVariable(bool isCommit); +extern void AtEOSubXact_SessionVariable(bool isCommit, + SubTransactionId mySubid, + SubTransactionId parentSubid); + +#endif diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index cc7b32b279..eecbbd7768 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -2130,6 +2130,7 @@ typedef enum ObjectType OBJECT_TSTEMPLATE, OBJECT_TYPE, OBJECT_USER_MAPPING, + OBJECT_VARIABLE, OBJECT_VIEW } ObjectType; @@ -3253,6 +3254,25 @@ typedef struct AlterStatsStmt bool missing_ok; /* skip error if statistics object is missing */ } AlterStatsStmt; + +/* ---------------------- + * {Create|Alter} VARIABLE Statement + * ---------------------- + */ +typedef struct CreateSessionVarStmt +{ + NodeTag type; + RangeVar *variable; /* the variable to create */ + TypeName *typeName; /* the type of variable */ + CollateClause *collClause; + Node *defexpr; /* default expression */ + char eoxaction; /* on commit action */ + bool if_not_exists; /* do nothing if it already exists */ + bool is_not_null; /* Disallow nulls */ + bool is_immutable; /* Don't allow changes */ +} CreateSessionVarStmt; + + /* ---------------------- * Create Function Statement * ---------------------- diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index f5b2e61ca5..9f79b90251 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -464,6 +464,8 @@ PG_KEYWORD("validator", VALIDATOR, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("value", VALUE_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("values", VALUES, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("varchar", VARCHAR, COL_NAME_KEYWORD, BARE_LABEL) +PG_KEYWORD("variable", VARIABLE, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("variables", VARIABLES, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("variadic", VARIADIC, RESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("varying", VARYING, UNRESERVED_KEYWORD, AS_LABEL) PG_KEYWORD("verbose", VERBOSE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h index f589112d5e..eccde83628 100644 --- a/src/include/parser/parse_node.h +++ b/src/include/parser/parse_node.h @@ -81,6 +81,7 @@ typedef enum ParseExprKind EXPR_KIND_COPY_WHERE, /* WHERE condition in COPY FROM */ EXPR_KIND_GENERATED_COLUMN, /* generation expression for a column */ EXPR_KIND_CYCLE_MARK, /* cycle mark value */ + EXPR_KIND_VARIABLE_DEFAULT, /* default value for session variable */ } ParseExprKind; diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h index e738ac1c09..259bdc994e 100644 --- a/src/include/tcop/cmdtaglist.h +++ b/src/include/tcop/cmdtaglist.h @@ -68,6 +68,7 @@ PG_CMDTAG(CMDTAG_ALTER_TRANSFORM, "ALTER TRANSFORM", true, false, false) PG_CMDTAG(CMDTAG_ALTER_TRIGGER, "ALTER TRIGGER", true, false, false) PG_CMDTAG(CMDTAG_ALTER_TYPE, "ALTER TYPE", true, true, false) PG_CMDTAG(CMDTAG_ALTER_USER_MAPPING, "ALTER USER MAPPING", true, false, false) +PG_CMDTAG(CMDTAG_ALTER_VARIABLE, "ALTER VARIABLE", true, false, false) PG_CMDTAG(CMDTAG_ALTER_VIEW, "ALTER VIEW", true, false, false) PG_CMDTAG(CMDTAG_ANALYZE, "ANALYZE", false, false, false) PG_CMDTAG(CMDTAG_BEGIN, "BEGIN", false, false, false) @@ -123,6 +124,7 @@ PG_CMDTAG(CMDTAG_CREATE_TRANSFORM, "CREATE TRANSFORM", true, false, false) PG_CMDTAG(CMDTAG_CREATE_TRIGGER, "CREATE TRIGGER", true, false, false) PG_CMDTAG(CMDTAG_CREATE_TYPE, "CREATE TYPE", true, false, false) PG_CMDTAG(CMDTAG_CREATE_USER_MAPPING, "CREATE USER MAPPING", true, false, false) +PG_CMDTAG(CMDTAG_CREATE_VARIABLE, "CREATE VARIABLE", true, false, false) PG_CMDTAG(CMDTAG_CREATE_VIEW, "CREATE VIEW", true, false, false) PG_CMDTAG(CMDTAG_DEALLOCATE, "DEALLOCATE", false, false, false) PG_CMDTAG(CMDTAG_DEALLOCATE_ALL, "DEALLOCATE ALL", false, false, false) @@ -175,6 +177,7 @@ PG_CMDTAG(CMDTAG_DROP_TRANSFORM, "DROP TRANSFORM", true, false, false) PG_CMDTAG(CMDTAG_DROP_TRIGGER, "DROP TRIGGER", true, false, false) PG_CMDTAG(CMDTAG_DROP_TYPE, "DROP TYPE", true, false, false) PG_CMDTAG(CMDTAG_DROP_USER_MAPPING, "DROP USER MAPPING", true, false, false) +PG_CMDTAG(CMDTAG_DROP_VARIABLE, "DROP VARIABLE", true, false, false) PG_CMDTAG(CMDTAG_DROP_VIEW, "DROP VIEW", true, false, false) PG_CMDTAG(CMDTAG_EXECUTE, "EXECUTE", false, false, false) PG_CMDTAG(CMDTAG_EXPLAIN, "EXPLAIN", false, false, false) diff --git a/src/include/utils/acl.h b/src/include/utils/acl.h index f8e1238fa2..42e286a7ae 100644 --- a/src/include/utils/acl.h +++ b/src/include/utils/acl.h @@ -169,6 +169,7 @@ typedef struct ArrayType Acl; #define ACL_ALL_RIGHTS_SCHEMA (ACL_USAGE|ACL_CREATE) #define ACL_ALL_RIGHTS_TABLESPACE (ACL_CREATE) #define ACL_ALL_RIGHTS_TYPE (ACL_USAGE) +#define ACL_ALL_RIGHTS_VARIABLE (ACL_SELECT|ACL_UPDATE) /* operation codes for pg_*_aclmask */ typedef enum diff --git a/src/include/utils/lsyscache.h b/src/include/utils/lsyscache.h index 4f5418b972..283db9f725 100644 --- a/src/include/utils/lsyscache.h +++ b/src/include/utils/lsyscache.h @@ -132,6 +132,7 @@ extern char get_func_prokind(Oid funcid); extern bool get_func_leakproof(Oid funcid); extern RegProcedure get_func_support(Oid funcid); extern Oid get_relname_relid(const char *relname, Oid relnamespace); +extern Oid get_varname_varid(const char *varname, Oid varnamespace); extern char *get_rel_name(Oid relid); extern Oid get_rel_namespace(Oid relid); extern Oid get_rel_type_id(Oid relid); @@ -203,6 +204,14 @@ extern char *get_publication_name(Oid pubid, bool missing_ok); extern Oid get_subscription_oid(const char *subname, bool missing_ok); extern char *get_subscription_name(Oid subid, bool missing_ok); +extern char *get_session_variable_name(Oid varid); +extern Oid get_session_variable_namespace(Oid varid); +extern Oid get_session_variable_type(Oid varid); +extern void get_session_variable_type_typmod_collid(Oid varid, + Oid *typid, + int32 *typmod, + Oid *collid); + #define type_is_array(typid) (get_element_type(typid) != InvalidOid) /* type_is_array_domain accepts both plain arrays and domains over arrays */ #define type_is_array_domain(typid) (get_base_element_type(typid) != InvalidOid) diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h index 67ea6e4945..633c84c4d3 100644 --- a/src/include/utils/syscache.h +++ b/src/include/utils/syscache.h @@ -113,9 +113,11 @@ enum SysCacheIdentifier TYPENAMENSP, TYPEOID, USERMAPPINGOID, - USERMAPPINGUSERSERVER + USERMAPPINGUSERSERVER, + VARIABLENAMENSP, + VARIABLEOID -#define SysCacheSize (USERMAPPINGUSERSERVER + 1) +#define SysCacheSize (VARIABLEOID + 1) }; extern void InitCatalogCache(void); diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out index 215eb899be..d995332140 100644 --- a/src/test/regress/expected/oidjoins.out +++ b/src/test/regress/expected/oidjoins.out @@ -266,3 +266,7 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid} NOTICE: checking pg_subscription {subowner} => pg_authid {oid} NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid} NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid} +NOTICE: checking pg_variable {vartype} => pg_type {oid} +NOTICE: checking pg_variable {varnamespace} => pg_namespace {oid} +NOTICE: checking pg_variable {varowner} => pg_authid {oid} +NOTICE: checking pg_variable {varcollation} => pg_collation {oid} diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 06da7cd428..8c1df4b1fd 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -502,6 +502,7 @@ CreateRoleStmt CreateSchemaStmt CreateSchemaStmtContext CreateSeqStmt +CreateSessionVarStmt CreateStatsStmt CreateStmt CreateStmtContext @@ -830,6 +831,7 @@ FormData_pg_ts_parser FormData_pg_ts_template FormData_pg_type FormData_pg_user_mapping +FormData_pg_variable Form_pg_aggregate Form_pg_am Form_pg_amop @@ -888,6 +890,7 @@ Form_pg_ts_parser Form_pg_ts_template Form_pg_type Form_pg_user_mapping +Form_pg_variable FormatNode FreeBlockNumberArray FreeListData @@ -2664,6 +2667,8 @@ SupportRequestRows SupportRequestSelectivity SupportRequestSimplify SupportRequestWFuncMonotonic +SVariableXActAction +SVariableXActActionItem Syn SyncOps SyncRepConfigData -- 2.40.1 ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v1 2/7] Row pattern recognition patch (parse/analysis). @ 2023-06-25 11:48 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-06-25 11:48 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 196 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 209 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index f61f794755..5bb11add3c 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2949,6 +2952,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3814,3 +3821,190 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + if (wc->rpSkipTo != ST_NEXT_ROW) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("only AFTER MATCH SKIP TO NEXT_ROW is supported"))); + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static List * +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + List *patterns; + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return NULL; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } + return patterns; +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 346fd272b6..20231d9ec0 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -541,6 +541,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1754,6 +1755,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3133,6 +3135,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sun_Jun_25_21_05_09_2023_126)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v1-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v1 2/7] Row pattern recognition patch (parse/analysis). @ 2023-06-25 11:48 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-06-25 11:48 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 196 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 209 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index f61f794755..5bb11add3c 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2949,6 +2952,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3814,3 +3821,190 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + if (wc->rpSkipTo != ST_NEXT_ROW) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("only AFTER MATCH SKIP TO NEXT_ROW is supported"))); + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static List * +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + List *patterns; + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return NULL; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } + return patterns; +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 346fd272b6..20231d9ec0 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -541,6 +541,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1754,6 +1755,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3133,6 +3135,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sun_Jun_25_21_05_09_2023_126)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v1-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v1 2/7] Row pattern recognition patch (parse/analysis). @ 2023-06-25 11:48 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-06-25 11:48 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 196 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 209 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index f61f794755..5bb11add3c 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2949,6 +2952,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3814,3 +3821,190 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + if (wc->rpSkipTo != ST_NEXT_ROW) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("only AFTER MATCH SKIP TO NEXT_ROW is supported"))); + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static List * +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + List *patterns; + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return NULL; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } + return patterns; +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 346fd272b6..20231d9ec0 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -541,6 +541,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1754,6 +1755,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3133,6 +3135,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sun_Jun_25_21_05_09_2023_126)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v1-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v1 2/7] Row pattern recognition patch (parse/analysis). @ 2023-06-25 11:48 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-06-25 11:48 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 196 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 209 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index f61f794755..5bb11add3c 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2949,6 +2952,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3814,3 +3821,190 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + if (wc->rpSkipTo != ST_NEXT_ROW) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("only AFTER MATCH SKIP TO NEXT_ROW is supported"))); + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static List * +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + List *patterns; + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return NULL; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } + return patterns; +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 346fd272b6..20231d9ec0 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -541,6 +541,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1754,6 +1755,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3133,6 +3135,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sun_Jun_25_21_05_09_2023_126)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v1-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v1 2/7] Row pattern recognition patch (parse/analysis). @ 2023-06-25 11:48 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-06-25 11:48 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 196 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 209 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index f61f794755..5bb11add3c 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2949,6 +2952,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3814,3 +3821,190 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + if (wc->rpSkipTo != ST_NEXT_ROW) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("only AFTER MATCH SKIP TO NEXT_ROW is supported"))); + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static List * +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + List *patterns; + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return NULL; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } + return patterns; +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 346fd272b6..20231d9ec0 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -541,6 +541,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1754,6 +1755,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3133,6 +3135,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sun_Jun_25_21_05_09_2023_126)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v1-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v2 2/7] Row pattern recognition patch (parse/analysis). @ 2023-06-26 08:05 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-06-26 08:05 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 192 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 205 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index f61f794755..ccf3332bef 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2949,6 +2952,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3814,3 +3821,186 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static List * +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + List *patterns; + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return NULL; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } + return patterns; +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 346fd272b6..20231d9ec0 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -541,6 +541,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1754,6 +1755,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3133,6 +3135,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Jun_26_17_45_07_2023_724)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v2-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v2 2/7] Row pattern recognition patch (parse/analysis). @ 2023-06-26 08:05 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-06-26 08:05 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 192 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 205 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index f61f794755..ccf3332bef 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2949,6 +2952,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3814,3 +3821,186 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static List * +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + List *patterns; + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return NULL; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } + return patterns; +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 346fd272b6..20231d9ec0 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -541,6 +541,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1754,6 +1755,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3133,6 +3135,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Jun_26_17_45_07_2023_724)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v2-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v2 2/7] Row pattern recognition patch (parse/analysis). @ 2023-06-26 08:05 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-06-26 08:05 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 192 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 205 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index f61f794755..ccf3332bef 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2949,6 +2952,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3814,3 +3821,186 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static List * +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + List *patterns; + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return NULL; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } + return patterns; +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 346fd272b6..20231d9ec0 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -541,6 +541,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1754,6 +1755,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3133,6 +3135,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Jun_26_17_45_07_2023_724)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v2-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v2 2/7] Row pattern recognition patch (parse/analysis). @ 2023-06-26 08:05 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-06-26 08:05 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 192 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 205 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index f61f794755..ccf3332bef 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2949,6 +2952,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3814,3 +3821,186 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static List * +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + List *patterns; + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return NULL; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } + return patterns; +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 346fd272b6..20231d9ec0 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -541,6 +541,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1754,6 +1755,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3133,6 +3135,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Jun_26_17_45_07_2023_724)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v2-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v2 2/7] Row pattern recognition patch (parse/analysis). @ 2023-06-26 08:05 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-06-26 08:05 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 192 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 205 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index f61f794755..ccf3332bef 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2949,6 +2952,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3814,3 +3821,186 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static List * +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + List *patterns; + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return NULL; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } + return patterns; +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 346fd272b6..20231d9ec0 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -541,6 +541,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1754,6 +1755,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3133,6 +3135,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Jun_26_17_45_07_2023_724)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v2-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v3 2/7] Row pattern recognition patch (parse/analysis). @ 2023-07-26 10:49 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-07-26 10:49 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 190 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 203 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..ea2decc579 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,184 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index fed8e4d089..8921b7ae01 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Jul_26_21_21_34_2023_317)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v3-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v3 2/7] Row pattern recognition patch (parse/analysis). @ 2023-07-26 10:49 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-07-26 10:49 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 190 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 203 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..ea2decc579 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,184 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index fed8e4d089..8921b7ae01 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Jul_26_21_21_34_2023_317)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v3-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v3 2/7] Row pattern recognition patch (parse/analysis). @ 2023-07-26 10:49 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-07-26 10:49 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 190 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 203 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..ea2decc579 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,184 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index fed8e4d089..8921b7ae01 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Jul_26_21_21_34_2023_317)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v3-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v3 2/7] Row pattern recognition patch (parse/analysis). @ 2023-07-26 10:49 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-07-26 10:49 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 ++ src/backend/parser/parse_clause.c | 190 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 203 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..ea2decc579 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,184 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + /* Check Frame option. Frame must start at current row */ + + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + ResTarget *restarget, *r; + List *restargets; + + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char *name; + ListCell *l; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Make sure that row pattern definition search condition is a boolean + * expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index fed8e4d089..8921b7ae01 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Jul_26_21_21_34_2023_317)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v3-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v4 2/7] Row pattern recognition patch (parse/analysis). @ 2023-08-09 07:56 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-08-09 07:56 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index fed8e4d089..8921b7ae01 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Aug__9_17_41_12_2023_134)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v4-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v4 2/7] Row pattern recognition patch (parse/analysis). @ 2023-08-09 07:56 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-08-09 07:56 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index fed8e4d089..8921b7ae01 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Aug__9_17_41_12_2023_134)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v4-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v4 2/7] Row pattern recognition patch (parse/analysis). @ 2023-08-09 07:56 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-08-09 07:56 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index fed8e4d089..8921b7ae01 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Aug__9_17_41_12_2023_134)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v4-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v4 2/7] Row pattern recognition patch (parse/analysis). @ 2023-08-09 07:56 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-08-09 07:56 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index fed8e4d089..8921b7ae01 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Aug__9_17_41_12_2023_134)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v4-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v4 2/7] Row pattern recognition patch (parse/analysis). @ 2023-08-09 07:56 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-08-09 07:56 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index fed8e4d089..8921b7ae01 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Aug__9_17_41_12_2023_134)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v4-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v4 2/7] Row pattern recognition patch (parse/analysis). @ 2023-08-09 07:56 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-08-09 07:56 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index fed8e4d089..8921b7ae01 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Aug__9_17_41_12_2023_134)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v4-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-02 06:32 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-02 06:32 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sat_Sep__2_15_52_35_2023_273)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-02 06:32 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-02 06:32 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sat_Sep__2_15_52_35_2023_273)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-02 06:32 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-02 06:32 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sat_Sep__2_15_52_35_2023_273)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-02 06:32 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-02 06:32 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sat_Sep__2_15_52_35_2023_273)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-02 06:32 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-02 06:32 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sat_Sep__2_15_52_35_2023_273)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-02 06:32 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-02 06:32 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sat_Sep__2_15_52_35_2023_273)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-02 06:32 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-02 06:32 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sat_Sep__2_15_52_35_2023_273)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v5-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-12 05:22 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-12 05:22 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Tue_Sep_12_15_18_43_2023_359)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v6-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-12 05:22 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-12 05:22 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Tue_Sep_12_15_18_43_2023_359)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v6-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-12 05:22 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-12 05:22 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Tue_Sep_12_15_18_43_2023_359)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v6-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-12 05:22 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-12 05:22 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Tue_Sep_12_15_18_43_2023_359)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v6-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-12 05:22 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-12 05:22 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Tue_Sep_12_15_18_43_2023_359)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v6-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-12 05:22 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-12 05:22 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Tue_Sep_12_15_18_43_2023_359)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v6-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-12 05:22 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-12 05:22 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 292 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 305 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..60020a7025 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Tue_Sep_12_15_18_43_2023_359)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v6-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v7 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-22 04:53 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-22 04:53 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 293 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 306 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..293d4b1680 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Sep_22_14_16_40_2023_530)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v7-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v7 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-22 04:53 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-22 04:53 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 293 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 306 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..293d4b1680 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Sep_22_14_16_40_2023_530)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v7-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v7 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-22 04:53 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-22 04:53 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 293 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 306 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..293d4b1680 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Sep_22_14_16_40_2023_530)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v7-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v7 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-22 04:53 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-22 04:53 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 293 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 306 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..293d4b1680 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Sep_22_14_16_40_2023_530)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v7-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v7 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-22 04:53 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-22 04:53 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 293 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 306 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..293d4b1680 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Sep_22_14_16_40_2023_530)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v7-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v8 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-25 05:01 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-25 05:01 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 293 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 306 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..293d4b1680 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Sep_25_14_26_30_2023_752)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v8-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v8 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-25 05:01 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-25 05:01 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 293 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 306 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..293d4b1680 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Sep_25_14_26_30_2023_752)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v8-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v8 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-25 05:01 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-25 05:01 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 293 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 306 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..293d4b1680 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Sep_25_14_26_30_2023_752)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v8-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v8 2/7] Row pattern recognition patch (parse/analysis). @ 2023-09-25 05:01 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-09-25 05:01 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 293 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 306 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..293d4b1680 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Sep_25_14_26_30_2023_752)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v8-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v9 2/7] Row pattern recognition patch (parse/analysis). @ 2023-10-04 05:51 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-10-04 05:51 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 293 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 306 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..293d4b1680 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Oct__4_15_03_28_2023_821)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v9-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v9 2/7] Row pattern recognition patch (parse/analysis). @ 2023-10-04 05:51 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-10-04 05:51 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 293 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 306 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..293d4b1680 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Oct__4_15_03_28_2023_821)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v9-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v9 2/7] Row pattern recognition patch (parse/analysis). @ 2023-10-04 05:51 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-10-04 05:51 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 293 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 306 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..293d4b1680 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Oct__4_15_03_28_2023_821)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v9-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v9 2/7] Row pattern recognition patch (parse/analysis). @ 2023-10-04 05:51 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-10-04 05:51 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 293 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 306 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..293d4b1680 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + return transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Oct__4_15_03_28_2023_821)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v9-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v10 2/7] Row pattern recognition patch (parse/analysis). @ 2023-10-22 02:22 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-10-22 02:22 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 299 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..9c347216f7 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,293 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *)defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sun_Oct_22_11_39_20_2023_140)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v10-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v10 2/7] Row pattern recognition patch (parse/analysis). @ 2023-10-22 02:22 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-10-22 02:22 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 299 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..9c347216f7 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,293 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *)defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sun_Oct_22_11_39_20_2023_140)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v10-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v10 2/7] Row pattern recognition patch (parse/analysis). @ 2023-10-22 02:22 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-10-22 02:22 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 299 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..9c347216f7 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,293 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *)defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sun_Oct_22_11_39_20_2023_140)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v10-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v10 2/7] Row pattern recognition patch (parse/analysis). @ 2023-10-22 02:22 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-10-22 02:22 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 299 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 85cd47b7ae..aa7a1cee80 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -564,6 +564,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -953,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..9c347216f7 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,293 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *)defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc, *l; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* + * Primary row pattern variable names in PATTERN clause must appear in + * DEFINE clause as row pattern definition variable names. + */ + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + ResTarget *restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("primary row pattern variable name \"%s\" does not appear in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)a)))); + } + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index b3f0b6a137..2ff3699538 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sun_Oct_22_11_39_20_2023_140)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v10-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v11 2/7] Row pattern recognition patch (parse/analysis). @ 2023-11-08 06:57 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-11-08 06:57 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 278 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 291 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 9bbad33fbd..28fb5e0d71 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -575,6 +575,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -964,6 +968,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..c5d3c10683 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,272 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *)defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 6c29471bb3..086431f91b 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Nov__8_16_37_05_2023_872)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v11-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v11 2/7] Row pattern recognition patch (parse/analysis). @ 2023-11-08 06:57 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-11-08 06:57 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 278 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 291 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 9bbad33fbd..28fb5e0d71 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -575,6 +575,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -964,6 +968,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..c5d3c10683 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,272 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *)defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 6c29471bb3..086431f91b 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Nov__8_16_37_05_2023_872)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v11-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v11 2/7] Row pattern recognition patch (parse/analysis). @ 2023-11-08 06:57 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-11-08 06:57 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 278 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 291 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 9bbad33fbd..28fb5e0d71 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -575,6 +575,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -964,6 +968,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..c5d3c10683 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,272 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *)defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 6c29471bb3..086431f91b 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_Nov__8_16_37_05_2023_872)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v11-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v12 2/7] Row pattern recognition patch (parse/analysis). @ 2023-12-04 11:23 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-12-04 11:23 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 278 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 291 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 9bbad33fbd..28fb5e0d71 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -575,6 +575,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -964,6 +968,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..c5d3c10683 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,272 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *)defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 6c29471bb3..086431f91b 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Dec__8_10_16_13_2023_489)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v12-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v12 2/7] Row pattern recognition patch (parse/analysis). @ 2023-12-04 11:23 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-12-04 11:23 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 278 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 291 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 9bbad33fbd..28fb5e0d71 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -575,6 +575,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -964,6 +968,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..c5d3c10683 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,272 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *)defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 6c29471bb3..086431f91b 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Dec__8_10_16_13_2023_489)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v12-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v12 2/7] Row pattern recognition patch (parse/analysis). @ 2023-12-04 11:23 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2023-12-04 11:23 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 278 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 291 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 9bbad33fbd..28fb5e0d71 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -575,6 +575,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -964,6 +968,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 334b9b42bd..c5d3c10683 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,272 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *)defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 64c582c344..18b58ac263 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 6c29471bb3..086431f91b 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Dec__8_10_16_13_2023_489)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v12-0003-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v13 2/8] Row pattern recognition patch (parse/analysis). @ 2024-01-22 09:45 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-01-22 09:45 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 281 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 294 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 7b211a7743..5a9743be6e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -575,6 +575,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -964,6 +968,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4b50278fd0..104c0105c5 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,275 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *)defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 9300c7b9ab..da4f42677b 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index fdb3e6df33..bf07085a15 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Jan_22_19_26_18_2024_011)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v13-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v13 2/8] Row pattern recognition patch (parse/analysis). @ 2024-01-22 09:45 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-01-22 09:45 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 281 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 294 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 7b211a7743..5a9743be6e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -575,6 +575,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -964,6 +968,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4b50278fd0..104c0105c5 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,275 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *)defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 9300c7b9ab..da4f42677b 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index fdb3e6df33..bf07085a15 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Jan_22_19_26_18_2024_011)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v13-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v13 2/8] Row pattern recognition patch (parse/analysis). @ 2024-01-22 09:45 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-01-22 09:45 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 281 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 294 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 7b211a7743..5a9743be6e 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -575,6 +575,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -964,6 +968,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4b50278fd0..104c0105c5 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,10 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2953,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3822,275 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, *l; + ResTarget *restarget, *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. + * (the raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *)n; + restarget->location = -1; + restargets = lappend((List *)restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = list_concat(windef->rpCommonSyntax->rpDefs, + restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *)restarget->val, targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *)r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. + * We assign [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *)lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", initialLen), + parser_errposition(pstate, exprLocation((Node *)restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *)defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *)lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s","MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *)windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 9300c7b9ab..da4f42677b 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index fdb3e6df33..bf07085a15 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Jan_22_19_26_18_2024_011)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v13-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v14 2/8] Row pattern recognition patch (parse/analysis). @ 2024-02-28 13:59 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-02-28 13:59 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 9d151a880b..a36218b103 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -576,6 +576,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -965,6 +969,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4b50278fd0..8a4f8f24d2 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2957,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3826,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 9300c7b9ab..da4f42677b 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index fdb3e6df33..bf07085a15 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Feb_29_09_19_54_2024_640)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v14-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v14 2/8] Row pattern recognition patch (parse/analysis). @ 2024-02-28 13:59 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-02-28 13:59 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 9d151a880b..a36218b103 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -576,6 +576,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -965,6 +969,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4b50278fd0..8a4f8f24d2 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2957,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3826,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 9300c7b9ab..da4f42677b 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index fdb3e6df33..bf07085a15 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Feb_29_09_19_54_2024_640)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v14-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v14 2/8] Row pattern recognition patch (parse/analysis). @ 2024-02-28 13:59 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-02-28 13:59 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 9d151a880b..a36218b103 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -576,6 +576,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -965,6 +969,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4b50278fd0..8a4f8f24d2 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -100,7 +100,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2950,6 +2957,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3815,3 +3826,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 9300c7b9ab..da4f42677b 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -557,6 +557,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1770,6 +1771,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3149,6 +3151,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index fdb3e6df33..bf07085a15 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2656,6 +2656,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Feb_29_09_19_54_2024_640)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v14-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v15 2/8] Row pattern recognition patch (parse/analysis). @ 2024-03-28 10:30 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-03-28 10:30 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index d2ac86777c..de2e5791e3 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2948,6 +2955,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3813,3 +3824,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 73c83cea4a..8b0cc608bc 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 0cbc950c95..ad982a7c17 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2657,6 +2657,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Mar_28_19_59_25_2024_076)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v15-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v15 2/8] Row pattern recognition patch (parse/analysis). @ 2024-03-28 10:30 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-03-28 10:30 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index d2ac86777c..de2e5791e3 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2948,6 +2955,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3813,3 +3824,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 73c83cea4a..8b0cc608bc 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 0cbc950c95..ad982a7c17 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2657,6 +2657,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Mar_28_19_59_25_2024_076)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v15-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v15 2/8] Row pattern recognition patch (parse/analysis). @ 2024-03-28 10:30 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-03-28 10:30 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index d2ac86777c..de2e5791e3 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2948,6 +2955,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3813,3 +3824,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 73c83cea4a..8b0cc608bc 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 0cbc950c95..ad982a7c17 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2657,6 +2657,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Mar_28_19_59_25_2024_076)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v15-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v15 2/8] Row pattern recognition patch (parse/analysis). @ 2024-03-28 10:30 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-03-28 10:30 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index d2ac86777c..de2e5791e3 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2948,6 +2955,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3813,3 +3824,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 73c83cea4a..8b0cc608bc 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 0cbc950c95..ad982a7c17 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2657,6 +2657,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Mar_28_19_59_25_2024_076)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v15-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v15 2/8] Row pattern recognition patch (parse/analysis). @ 2024-03-28 10:30 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-03-28 10:30 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index d2ac86777c..de2e5791e3 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2948,6 +2955,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3813,3 +3824,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 73c83cea4a..8b0cc608bc 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 0cbc950c95..ad982a7c17 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2657,6 +2657,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Mar_28_19_59_25_2024_076)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v15-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v16 2/8] Row pattern recognition patch (parse/analysis). @ 2024-04-12 06:49 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-04-12 06:49 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4fc5fc87e0..003a1e14ce 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3821,3 +3832,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 4c98d7a046..a9f9d47854 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 0cbc950c95..ad982a7c17 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2657,6 +2657,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Apr_12_16_09_08_2024_262)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v16-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v16 2/8] Row pattern recognition patch (parse/analysis). @ 2024-04-12 06:49 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-04-12 06:49 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4fc5fc87e0..003a1e14ce 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3821,3 +3832,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 4c98d7a046..a9f9d47854 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 0cbc950c95..ad982a7c17 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2657,6 +2657,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Apr_12_16_09_08_2024_262)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v16-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v16 2/8] Row pattern recognition patch (parse/analysis). @ 2024-04-12 06:49 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-04-12 06:49 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4fc5fc87e0..003a1e14ce 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3821,3 +3832,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 4c98d7a046..a9f9d47854 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 0cbc950c95..ad982a7c17 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2657,6 +2657,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Apr_12_16_09_08_2024_262)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v16-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v17 2/8] Row pattern recognition patch (parse/analysis). @ 2024-04-28 11:00 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-04-28 11:00 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4fc5fc87e0..003a1e14ce 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3821,3 +3832,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 1c1c86aa3e..5540a0fb0a 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 0cbc950c95..ad982a7c17 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2657,6 +2657,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sun_Apr_28_20_28_26_2024_444)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v17-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v17 2/8] Row pattern recognition patch (parse/analysis). @ 2024-04-28 11:00 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-04-28 11:00 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4fc5fc87e0..003a1e14ce 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3821,3 +3832,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 1c1c86aa3e..5540a0fb0a 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 0cbc950c95..ad982a7c17 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2657,6 +2657,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sun_Apr_28_20_28_26_2024_444)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v17-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v17 2/8] Row pattern recognition patch (parse/analysis). @ 2024-04-28 11:00 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-04-28 11:00 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4fc5fc87e0..003a1e14ce 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->runCondition = NIL; wc->winref = winref; @@ -3821,3 +3832,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 1c1c86aa3e..5540a0fb0a 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 0cbc950c95..ad982a7c17 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2657,6 +2657,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sun_Apr_28_20_28_26_2024_444)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v17-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v18 2/8] Row pattern recognition patch (parse/analysis). @ 2024-05-11 07:11 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-05-11 07:11 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index aba3546ed1..eb138087bf 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sat_May_11_16_23_07_2024_789)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v18-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v18 2/8] Row pattern recognition patch (parse/analysis). @ 2024-05-11 07:11 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-05-11 07:11 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index aba3546ed1..eb138087bf 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sat_May_11_16_23_07_2024_789)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v18-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v18 2/8] Row pattern recognition patch (parse/analysis). @ 2024-05-11 07:11 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-05-11 07:11 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 4 + src/backend/parser/parse_func.c | 3 + 4 files changed, 309 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index aba3546ed1..eb138087bf 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1817,6 +1818,7 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_VALUES: case EXPR_KIND_VALUES_SINGLE: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; case EXPR_KIND_CHECK_CONSTRAINT: @@ -3197,6 +3199,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sat_May_11_16_23_07_2024_789)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v18-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v19 2/8] Row pattern recognition patch (parse/analysis). @ 2024-05-14 23:26 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-05-14 23:26 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 311 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index aba3546ed1..e98b45e06e 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_May_15_09_02_03_2024_008)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v19-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v19 2/8] Row pattern recognition patch (parse/analysis). @ 2024-05-14 23:26 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-05-14 23:26 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 311 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index aba3546ed1..e98b45e06e 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_May_15_09_02_03_2024_008)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v19-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v19 2/8] Row pattern recognition patch (parse/analysis). @ 2024-05-14 23:26 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-05-14 23:26 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 311 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index aba3546ed1..e98b45e06e 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Wed_May_15_09_02_03_2024_008)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v19-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v20 2/8] Row pattern recognition patch (parse/analysis). @ 2024-05-24 02:26 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-05-24 02:26 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 311 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index aba3546ed1..e98b45e06e 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_May_24_11_39_19_2024_763)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v20-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v20 2/8] Row pattern recognition patch (parse/analysis). @ 2024-05-24 02:26 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-05-24 02:26 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 311 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index aba3546ed1..e98b45e06e 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_May_24_11_39_19_2024_763)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v20-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v20 2/8] Row pattern recognition patch (parse/analysis). @ 2024-05-24 02:26 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-05-24 02:26 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 311 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index aba3546ed1..e98b45e06e 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -578,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_May_24_11_39_19_2024_763)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v20-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v21 2/8] Row pattern recognition patch (parse/analysis). @ 2024-08-26 04:32 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-08-26 04:32 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 311 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 56e413da9f..c187b3278d 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -577,6 +577,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3199,6 +3203,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Aug_26_13_39_47_2024_878)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v21-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v21 2/8] Row pattern recognition patch (parse/analysis). @ 2024-08-26 04:32 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-08-26 04:32 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 311 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 56e413da9f..c187b3278d 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -577,6 +577,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3199,6 +3203,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Aug_26_13_39_47_2024_878)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v21-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v21 2/8] Row pattern recognition patch (parse/analysis). @ 2024-08-26 04:32 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-08-26 04:32 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 311 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index bee7d8346a..9bc22a836a 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -577,6 +577,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -967,6 +971,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 56e413da9f..c187b3278d 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -577,6 +577,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3199,6 +3203,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Aug_26_13_39_47_2024_878)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v21-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v22 2/8] Row pattern recognition patch (parse/analysis). @ 2024-09-19 04:48 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-09-19 04:48 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 311 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index efa730c167..a80263f90d 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 36c1b7a88f..fe154bcaa0 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -577,6 +577,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3199,6 +3203,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Sep_19_13_59_47_2024_608)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v22-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v22 2/8] Row pattern recognition patch (parse/analysis). @ 2024-09-19 04:48 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-09-19 04:48 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 311 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index efa730c167..a80263f90d 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 36c1b7a88f..fe154bcaa0 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -577,6 +577,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3199,6 +3203,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Sep_19_13_59_47_2024_608)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v22-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v22 2/8] Row pattern recognition patch (parse/analysis). @ 2024-09-19 04:48 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-09-19 04:48 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 296 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 311 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index efa730c167..a80263f90d 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 8118036495..9762dce81f 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3820,3 +3831,286 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause Process DEFINE clause and transform ResTarget into + * list of TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 36c1b7a88f..fe154bcaa0 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -577,6 +577,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3199,6 +3203,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Sep_19_13_59_47_2024_608)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v22-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v23 2/8] Row pattern recognition patch (parse/analysis). @ 2024-10-25 03:56 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-10-25 03:56 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index efa730c167..a80263f90d 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4c97690908..a6d2f7f4ba 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3823,3 +3834,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index ef0b560f5e..b30b9f2675 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -577,6 +577,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3199,6 +3203,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Oct_25_13_04_53_2024_648)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v23-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v23 2/8] Row pattern recognition patch (parse/analysis). @ 2024-10-25 03:56 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-10-25 03:56 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index efa730c167..a80263f90d 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4c97690908..a6d2f7f4ba 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3823,3 +3834,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index ef0b560f5e..b30b9f2675 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -577,6 +577,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3199,6 +3203,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Oct_25_13_04_53_2024_648)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v23-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v23 2/8] Row pattern recognition patch (parse/analysis). @ 2024-10-25 03:56 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-10-25 03:56 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index efa730c167..a80263f90d 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 4c97690908..a6d2f7f4ba 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -98,7 +98,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2956,6 +2963,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3823,3 +3834,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index ef0b560f5e..b30b9f2675 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -577,6 +577,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1860,6 +1861,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3199,6 +3203,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Fri_Oct_25_13_04_53_2024_648)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v23-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v24 2/8] Row pattern recognition patch (parse/analysis). @ 2024-12-19 06:06 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-12-19 06:06 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 04b4596a65..6af3bcb375 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 979926b605..5a44c68b08 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -96,7 +96,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2954,6 +2961,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3821,3 +3832,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index c2806297aa..e827c59fd4 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -575,6 +575,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1858,6 +1859,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Dec_19_15_19_50_2024_894)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v24-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v24 2/8] Row pattern recognition patch (parse/analysis). @ 2024-12-19 06:06 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-12-19 06:06 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 04b4596a65..6af3bcb375 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 979926b605..5a44c68b08 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -96,7 +96,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2954,6 +2961,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3821,3 +3832,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index c2806297aa..e827c59fd4 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -575,6 +575,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1858,6 +1859,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Dec_19_15_19_50_2024_894)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v24-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v24 2/8] Row pattern recognition patch (parse/analysis). @ 2024-12-19 06:06 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-12-19 06:06 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 04b4596a65..6af3bcb375 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 979926b605..5a44c68b08 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -96,7 +96,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2954,6 +2961,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3821,3 +3832,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index c2806297aa..e827c59fd4 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -575,6 +575,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1858,6 +1859,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Thu_Dec_19_15_19_50_2024_894)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v24-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v25 2/9] Row pattern recognition patch (parse/analysis). @ 2024-12-21 06:19 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-12-21 06:19 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 04b4596a65..6af3bcb375 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 979926b605..5a44c68b08 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -96,7 +96,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2954,6 +2961,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3821,3 +3832,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index c2806297aa..e827c59fd4 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -575,6 +575,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1858,6 +1859,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sat_Dec_21_18_20_04_2024_526)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v25-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v25 2/9] Row pattern recognition patch (parse/analysis). @ 2024-12-21 06:19 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-12-21 06:19 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 04b4596a65..6af3bcb375 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 979926b605..5a44c68b08 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -96,7 +96,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2954,6 +2961,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3821,3 +3832,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index c2806297aa..e827c59fd4 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -575,6 +575,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1858,6 +1859,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sat_Dec_21_18_20_04_2024_526)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v25-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v25 2/9] Row pattern recognition patch (parse/analysis). @ 2024-12-21 06:19 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-12-21 06:19 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 04b4596a65..6af3bcb375 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 979926b605..5a44c68b08 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -96,7 +96,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2954,6 +2961,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3821,3 +3832,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index c2806297aa..e827c59fd4 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -575,6 +575,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1858,6 +1859,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Sat_Dec_21_18_20_04_2024_526)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v25-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v26 2/9] Row pattern recognition patch (parse/analysis). @ 2024-12-30 12:44 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-12-30 12:44 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 04b4596a65..6af3bcb375 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 979926b605..5a44c68b08 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -96,7 +96,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2954,6 +2961,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3821,3 +3832,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index c2806297aa..e827c59fd4 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -575,6 +575,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1858,6 +1859,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Dec_30_22_37_18_2024_171)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v26-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v26 2/9] Row pattern recognition patch (parse/analysis). @ 2024-12-30 12:44 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-12-30 12:44 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 04b4596a65..6af3bcb375 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 979926b605..5a44c68b08 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -96,7 +96,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2954,6 +2961,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3821,3 +3832,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index c2806297aa..e827c59fd4 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -575,6 +575,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1858,6 +1859,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Dec_30_22_37_18_2024_171)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v26-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v26 2/9] Row pattern recognition patch (parse/analysis). @ 2024-12-30 12:44 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-12-30 12:44 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 04b4596a65..6af3bcb375 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 979926b605..5a44c68b08 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -96,7 +96,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2954,6 +2961,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3821,3 +3832,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index c2806297aa..e827c59fd4 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -575,6 +575,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1858,6 +1859,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Mon_Dec_30_22_37_18_2024_171)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v26-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v27 2/9] Row pattern recognition patch (parse/analysis). @ 2024-12-30 23:53 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-12-30 23:53 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 04b4596a65..6af3bcb375 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 979926b605..5a44c68b08 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -96,7 +96,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2954,6 +2961,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3821,3 +3832,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index c2806297aa..e827c59fd4 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -575,6 +575,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1858,6 +1859,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Tue_Dec_31_08_57_07_2024_963)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v27-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v27 2/9] Row pattern recognition patch (parse/analysis). @ 2024-12-30 23:53 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-12-30 23:53 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 04b4596a65..6af3bcb375 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 979926b605..5a44c68b08 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -96,7 +96,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2954,6 +2961,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3821,3 +3832,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index c2806297aa..e827c59fd4 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -575,6 +575,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1858,6 +1859,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Tue_Dec_31_08_57_07_2024_963)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v27-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v27 2/9] Row pattern recognition patch (parse/analysis). @ 2024-12-30 23:53 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Tatsuo Ishii @ 2024-12-30 23:53 UTC (permalink / raw) --- src/backend/parser/parse_agg.c | 7 + src/backend/parser/parse_clause.c | 297 +++++++++++++++++++++++++++++- src/backend/parser/parse_expr.c | 6 + src/backend/parser/parse_func.c | 3 + 4 files changed, 312 insertions(+), 1 deletion(-) diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c index 04b4596a65..6af3bcb375 100644 --- a/src/backend/parser/parse_agg.c +++ b/src/backend/parser/parse_agg.c @@ -580,6 +580,10 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr) errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; + /* * There is intentionally no default: case here, so that the * compiler will warn if we add a new ParseExprKind without @@ -970,6 +974,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc, case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_clause.c b/src/backend/parser/parse_clause.c index 979926b605..5a44c68b08 100644 --- a/src/backend/parser/parse_clause.c +++ b/src/backend/parser/parse_clause.c @@ -96,7 +96,14 @@ static WindowClause *findWindowClause(List *wclist, const char *name); static Node *transformFrameOffset(ParseState *pstate, int frameOptions, Oid rangeopfamily, Oid rangeopcintype, Oid *inRangeFunc, Node *clause); - +static void transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static List *transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist); +static void transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); +static List *transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef); /* * transformFromClause - @@ -2954,6 +2961,10 @@ transformWindowDefinitions(ParseState *pstate, rangeopfamily, rangeopcintype, &wc->endInRangeFunc, windef->endOffset); + + /* Process Row Pattern Recognition related clauses */ + transformRPR(pstate, wc, windef, targetlist); + wc->winref = winref; result = lappend(result, wc); @@ -3821,3 +3832,287 @@ transformFrameOffset(ParseState *pstate, int frameOptions, return node; } + +/* + * transformRPR + * Process Row Pattern Recognition related clauses + */ +static void +transformRPR(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* + * Window definition exists? + */ + if (windef == NULL) + return; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + /* Check Frame option. Frame must start at current row */ + if ((wc->frameOptions & FRAMEOPTION_START_CURRENT_ROW) == 0) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("FRAME must start at current row when row patttern recognition is used"))); + + /* Transform AFTER MACH SKIP TO clause */ + wc->rpSkipTo = windef->rpCommonSyntax->rpSkipTo; + + /* Transform AFTER MACH SKIP TO variable */ + wc->rpSkipVariable = windef->rpCommonSyntax->rpSkipVariable; + + /* Transform SEEK or INITIAL clause */ + wc->initial = windef->rpCommonSyntax->initial; + + /* Transform DEFINE clause into list of TargetEntry's */ + wc->defineClause = transformDefineClause(pstate, wc, windef, targetlist); + + /* Check PATTERN clause and copy to patternClause */ + transformPatternClause(pstate, wc, windef); + + /* Transform MEASURE clause */ + transformMeasureClause(pstate, wc, windef); +} + +/* + * transformDefineClause + * Process DEFINE clause and transform ResTarget into list of + * TargetEntry. + * + * XXX we only support column reference in row pattern definition search + * condition, e.g. "price". <row pattern definition variable name>.<column + * reference> is not supported, e.g. "A.price". + */ +static List * +transformDefineClause(ParseState *pstate, WindowClause *wc, WindowDef *windef, + List **targetlist) +{ + /* DEFINE variable name initials */ + static char *defineVariableInitials = "abcdefghijklmnopqrstuvwxyz"; + + ListCell *lc, + *l; + ResTarget *restarget, + *r; + List *restargets; + List *defineClause; + char *name; + int initialLen; + int i; + + /* + * If Row Definition Common Syntax exists, DEFINE clause must exist. (the + * raw parser should have already checked it.) + */ + Assert(windef->rpCommonSyntax->rpDefs != NULL); + + /* + * Check and add "A AS A IS TRUE" if pattern variable is missing in DEFINE + * per the SQL standard. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + bool found = false; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + foreach(l, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(l); + + if (!strcmp(restarget->name, name)) + { + found = true; + break; + } + } + + if (!found) + { + /* + * "name" is missing. So create "name AS name IS TRUE" ResTarget + * node and add it to the temporary list. + */ + A_Const *n; + + restarget = makeNode(ResTarget); + n = makeNode(A_Const); + n->val.boolval.type = T_Boolean; + n->val.boolval.boolval = true; + n->location = -1; + restarget->name = pstrdup(name); + restarget->indirection = NIL; + restarget->val = (Node *) n; + restarget->location = -1; + restargets = lappend((List *) restargets, restarget); + } + } + + if (list_length(restargets) >= 1) + { + /* add missing DEFINEs */ + windef->rpCommonSyntax->rpDefs = + list_concat(windef->rpCommonSyntax->rpDefs, restargets); + list_free(restargets); + } + + /* + * Check for duplicate row pattern definition variables. The standard + * requires that no two row pattern definition variable names shall be + * equivalent. + */ + restargets = NIL; + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + /* + * Add DEFINE expression (Restarget->val) to the targetlist as a + * TargetEntry if it does not exist yet. Planner will add the column + * ref var node to the outer plan's target list later on. This makes + * DEFINE expression could access the outer tuple while evaluating + * PATTERN. + * + * XXX: adding whole expressions of DEFINE to the plan.targetlist is + * not so good, because it's not necessary to evalute the expression + * in the target list while running the plan. We should extract the + * var nodes only then add them to the plan.targetlist. + */ + findTargetlistEntrySQL99(pstate, (Node *) restarget->val, + targetlist, EXPR_KIND_RPR_DEFINE); + + /* + * Make sure that the row pattern definition search condition is a + * boolean expression. + */ + transformWhereClause(pstate, restarget->val, + EXPR_KIND_RPR_DEFINE, "DEFINE"); + + foreach(l, restargets) + { + char *n; + + r = (ResTarget *) lfirst(l); + n = r->name; + + if (!strcmp(n, name)) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("row pattern definition variable name \"%s\" appears more than once in DEFINE clause", + name), + parser_errposition(pstate, exprLocation((Node *) r)))); + } + restargets = lappend(restargets, restarget); + } + list_free(restargets); + + /* + * Create list of row pattern DEFINE variable name's initial. We assign + * [a-z] to them (up to 26 variable names are allowed). + */ + restargets = NIL; + i = 0; + initialLen = strlen(defineVariableInitials); + + foreach(lc, windef->rpCommonSyntax->rpDefs) + { + char initial[2]; + + restarget = (ResTarget *) lfirst(lc); + name = restarget->name; + + if (i >= initialLen) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("number of row pattern definition variable names exceeds %d", + initialLen), + parser_errposition(pstate, + exprLocation((Node *) restarget)))); + } + initial[0] = defineVariableInitials[i++]; + initial[1] = '\0'; + wc->defineInitial = lappend(wc->defineInitial, + makeString(pstrdup(initial))); + } + + defineClause = transformTargetList(pstate, windef->rpCommonSyntax->rpDefs, + EXPR_KIND_RPR_DEFINE); + + /* mark column origins */ + markTargetListOrigins(pstate, defineClause); + + /* mark all nodes in the DEFINE clause tree with collation information */ + assign_expr_collations(pstate, (Node *) defineClause); + + return defineClause; +} + +/* + * transformPatternClause + * Process PATTERN clause and return PATTERN clause in the raw parse tree + */ +static void +transformPatternClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + ListCell *lc; + + /* + * Row Pattern Common Syntax clause exists? + */ + if (windef->rpCommonSyntax == NULL) + return; + + wc->patternVariable = NIL; + wc->patternRegexp = NIL; + foreach(lc, windef->rpCommonSyntax->rpPatterns) + { + A_Expr *a; + char *name; + char *regexp; + + if (!IsA(lfirst(lc), A_Expr)) + ereport(ERROR, + errmsg("node type is not A_Expr")); + + a = (A_Expr *) lfirst(lc); + name = strVal(a->lexpr); + + wc->patternVariable = lappend(wc->patternVariable, makeString(pstrdup(name))); + regexp = strVal(lfirst(list_head(a->name))); + + wc->patternRegexp = lappend(wc->patternRegexp, makeString(pstrdup(regexp))); + } +} + +/* + * transformMeasureClause + * Process MEASURE clause + * XXX MEASURE clause is not supported yet + */ +static List * +transformMeasureClause(ParseState *pstate, WindowClause *wc, + WindowDef *windef) +{ + if (windef->rowPatternMeasures == NIL) + return NIL; + + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("%s", "MEASURE clause is not supported yet"), + parser_errposition(pstate, exprLocation((Node *) windef->rowPatternMeasures)))); + return NIL; +} diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index c2806297aa..e827c59fd4 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -575,6 +575,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref) case EXPR_KIND_COPY_WHERE: case EXPR_KIND_GENERATED_COLUMN: case EXPR_KIND_CYCLE_MARK: + case EXPR_KIND_RPR_DEFINE: /* okay */ break; @@ -1858,6 +1859,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink) case EXPR_KIND_GENERATED_COLUMN: err = _("cannot use subquery in column generation expression"); break; + case EXPR_KIND_RPR_DEFINE: + err = _("cannot use subquery in DEFINE expression"); + break; /* * There is intentionally no default: case here, so that the @@ -3197,6 +3201,8 @@ ParseExprKindName(ParseExprKind exprKind) return "GENERATED AS"; case EXPR_KIND_CYCLE_MARK: return "CYCLE"; + case EXPR_KIND_RPR_DEFINE: + return "DEFINE"; /* * There is intentionally no default: case here, so that the diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c index 9b23344a3b..4c482abb30 100644 --- a/src/backend/parser/parse_func.c +++ b/src/backend/parser/parse_func.c @@ -2658,6 +2658,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location) case EXPR_KIND_CYCLE_MARK: errkind = true; break; + case EXPR_KIND_RPR_DEFINE: + errkind = true; + break; /* * There is intentionally no default: case here, so that the -- 2.25.1 ----Next_Part(Tue_Dec_31_08_57_07_2024_963)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v27-0003-Row-pattern-recognition-patch-rewriter.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v3 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - This relies on the existing PGSTAT_MIN_INTERVAL to fire every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/tcop/postgres.c | 16 ++++ src/backend/utils/activity/pgstat.c | 111 +++++++++++++++++++++++----- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 15 ++++ src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 16 ++++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 9 files changed, 148 insertions(+), 18 deletions(-) 8.1% src/backend/tcop/ 68.4% src/backend/utils/activity/ 9.6% src/backend/utils/init/ 9.2% src/include/utils/ 4.1% src/include/ diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e54bf1e760f..132fae61423 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,22 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + + /* Schedule next timeout */ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_MIN_INTERVAL); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..ab4d9088a9a 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -122,8 +122,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +185,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +287,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +305,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +322,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +338,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +356,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +374,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +393,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +410,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +427,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +444,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +462,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +480,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +786,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1345,9 +1344,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1381,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1402,6 +1420,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2164,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* + * Exit if no pending stats at all. This avoids unnecessary work when + * backends are idle or in sessions without stats accumulation. + * + * Note: This check isn't precise as there might be only transactional + * stats pending, which we'll skip during the flush. However, maintaining + * precise tracking would add complexity that does not seem worth it from + * a performance point of view (no noticeable performance regression has + * been observed with the current implementation). + */ + if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed) + return; + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..6076f531c4a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void); static void IdleSessionTimeoutHandler(void); static void IdleStatsUpdateTimeoutHandler(void); static void ClientCheckTimeoutHandler(void); +static void AnytimeStatsUpdateTimeoutHandler(void); static bool ThereIsAtLeastOneRole(void); static void process_startup_options(Port *port, bool am_superuser); static void process_settings(Oid databaseid, Oid roleid); @@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); } /* @@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void) return result; } + +/* + * Timeout handler for flushing non-transactional stats. + */ +static void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..1651f16f966 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..46ce90c9624 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,9 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* Flush mode */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 3f3a888fd0e..d3912b43fdc 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2268,6 +2268,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --lCc1MWfC7i2uqs8g Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v3-0002-Remove-useless-calls-to-flush-some-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v2 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushBehavior enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - Register ANYTIME_STATS_UPDATE_TIMEOUT that fires every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/tcop/postgres.c | 16 ++++ src/backend/utils/activity/pgstat.c | 113 ++++++++++++++++++++++++---- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 15 ++++ src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 13 ++++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 9 files changed, 149 insertions(+), 16 deletions(-) 8.2% src/backend/tcop/ 69.3% src/backend/utils/activity/ 9.7% src/backend/utils/init/ 7.7% src/include/utils/ 4.5% src/include/ diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e54bf1e760f..9c4a9078ee0 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,22 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + + /* Schedule next timeout */ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_ANYTIME_FLUSH_INTERVAL); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..0f45a7d165e 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +289,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +307,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +324,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +340,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +358,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +376,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_behavior = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +395,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +412,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +429,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +446,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +464,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +482,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +788,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1345,9 +1346,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1383,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1397,11 +1417,42 @@ pgstat_flush_pending_entries(bool nowait) cur = next; } + /* + * When in anytime_only mode, the list may not be empty because + * FLUSH_AT_TXN_BOUNDARY entries were skipped. + */ Assert(dlist_is_empty(&pgStatPending) == !have_pending); return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2170,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* + * Exit if no pending stats at all. This avoids unnecessary work when + * backends are idle or in sessions without stats accumulation. + * + * Note: This check isn't precise as there might be only transactional + * stats pending, which we'll skip during the flush. However, maintaining + * precise tracking would add complexity that does not seem worth it from + * a performance point of view (no noticeable performance regression has + * been observed with the current implementation). + */ + if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed) + return; + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..cb0f6aecad1 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void); static void IdleSessionTimeoutHandler(void); static void IdleStatsUpdateTimeoutHandler(void); static void ClientCheckTimeoutHandler(void); +static void AnytimeStatsUpdateTimeoutHandler(void); static bool ThereIsAtLeastOneRole(void); static void process_startup_options(Port *port, bool am_superuser); static void process_settings(Oid databaseid, Oid roleid); @@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_ANYTIME_FLUSH_INTERVAL); } /* @@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void) return result; } + +/* + * Timeout handler for flushing non-transactional stats. + */ +static void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..86e65397614 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* When to call pgstat_report_anytime_stat() again */ +#define PGSTAT_ANYTIME_FLUSH_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..63feae640d1 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,16 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush behavior for statistics kinds. + */ +typedef enum PgStat_FlushBehavior +{ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ +} PgStat_FlushBehavior; /* * Metadata for a specific kind of statistics. @@ -251,6 +261,9 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* Flush behavior */ + PgStat_FlushBehavior flush_behavior; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 3f3a888fd0e..610b35a9b31 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2268,6 +2268,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushBehavior PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --OFsrIl+bjhifp5hk Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v2-0002-Remove-useless-calls-to-flush-some-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v1 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushBehavior enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - Register ANYTIME_STATS_UPDATE_TIMEOUT that fires every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/tcop/postgres.c | 18 +++++ src/backend/utils/activity/pgstat.c | 119 ++++++++++++++++++++++++---- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 15 ++++ src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 11 +++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 9 files changed, 154 insertions(+), 17 deletions(-) 9.7% src/backend/tcop/ 70.2% src/backend/utils/activity/ 9.3% src/backend/utils/init/ 6.0% src/include/utils/ 4.3% src/include/ diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e54bf1e760f..6a91543f80a 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,24 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + /* Skip if completely idle */ + if (!DoingCommandRead || IsTransactionOrTransactionBlock()) + pgstat_report_anytime_stat(false); + + /* Schedule next timeout */ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_ANYTIME_FLUSH_INTERVAL); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..f7942e47475 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +289,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +307,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +324,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +340,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +358,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +376,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_behavior = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +395,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +412,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +429,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +446,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +464,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +482,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +788,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1345,9 +1346,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1383,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1397,11 +1417,42 @@ pgstat_flush_pending_entries(bool nowait) cur = next; } - Assert(dlist_is_empty(&pgStatPending) == !have_pending); + /* + * When in anytime_only mode, the list may not be empty because + * FLUSH_AT_TXN_BOUNDARY entries were skipped. + */ + Assert(!anytime_only || dlist_is_empty(&pgStatPending) == !have_pending); return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2170,37 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flush non-transactional stats + * + * This is safe to call even inside a transaction. It only flushes stats + * kinds marked as FLUSH_ANYTIME. + * + * This allows long running transactions to report activity without waiting + * for transaction to finish. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* + * Exit if no pending stats at all. This avoids unnecessary work when + * backends are idle or in sessions without stats accumulation. + * + * Note: This check isn't precise as there might be only transactional + * stats pending, which we'll skip during the flush. However, maintaining + * precise tracking would add complexity that does not seem worth it from + * a performance point of view (no noticeable performance regression has + * been observed with the current implementation). + */ + if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed) + return; + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..cb0f6aecad1 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void); static void IdleSessionTimeoutHandler(void); static void IdleStatsUpdateTimeoutHandler(void); static void ClientCheckTimeoutHandler(void); +static void AnytimeStatsUpdateTimeoutHandler(void); static bool ThereIsAtLeastOneRole(void); static void process_startup_options(Port *port, bool am_superuser); static void process_settings(Oid databaseid, Oid roleid); @@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_ANYTIME_FLUSH_INTERVAL); } /* @@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void) return result; } + +/* + * Timeout handler for flushing non-transactional stats. + */ +static void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..86e65397614 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* When to call pgstat_report_anytime_stat() again */ +#define PGSTAT_ANYTIME_FLUSH_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..02f4f13fc0f 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,14 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush behavior for statistics kinds. + */ +typedef enum PgStat_FlushBehavior +{ + FLUSH_ANYTIME, /* All fields can flush anytime */ + FLUSH_AT_TXN_BOUNDARY, /* All fields need transaction boundary */ +} PgStat_FlushBehavior; /* * Metadata for a specific kind of statistics. @@ -251,6 +259,9 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* Flush behavior */ + PgStat_FlushBehavior flush_behavior; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 09e7f1d420e..9aabb325f16 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2261,6 +2261,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushBehavior PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --tjEWjIIwfNIHQLgt Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v1-0002-Remove-useless-calls-to-flush-some-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v11 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 121 +++++++++++++++--- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 22 ++++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 279 insertions(+), 66 deletions(-) 10.5% src/backend/postmaster/ 5.8% src/backend/replication/ 51.0% src/backend/utils/activity/ 5.8% src/backend/ 18.7% src/include/utils/ 6.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13cce9b49f1..cf29fc91f70 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 742137edad6..f1bae9d23d6 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -246,7 +248,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -268,6 +270,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 7c0e2809c17..bcf59227a00 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -103,7 +105,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -113,6 +115,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 7c1b8757d7d..aecc7a127e6 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index d01a09dd0c4..8c30efa2443 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3564,6 +3564,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..ddd331e2c81 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -108,10 +108,12 @@ #include "pgstat.h" #include "storage/fd.h" #include "storage/ipc.h" +#include "storage/latch.h" #include "storage/lwlock.h" #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +124,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -218,6 +219,12 @@ PgStat_LocalState pgStatLocal; */ bool pgstat_report_fixed = false; +/* + * Track when there is pending anytime flush to avoid relying on + * get_timeout_active() in hot pathes. + */ +bool pgstat_pending_anytime = false; + /* ---------- * Local data * @@ -288,6 +295,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +313,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +330,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +346,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +364,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +382,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +449,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +467,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +485,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +791,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1297,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1350,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1387,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1426,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2170,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); + + pgstat_pending_anytime = false; +} + +/* + * Timeout handler for flushing anytime stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index f2f8d3ff75f..b09316d3ab3 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 933dcb5cae5..8e86df60461 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 3277cf88a4e..6b6eec7578d 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -117,11 +117,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..2c2f3f10e10 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index b59e08605cc..eeeac1bf39a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index f16f35659b9..84e698da214 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 9bb777c3d5a..b011a315679 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -34,6 +34,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -532,8 +535,24 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !pgstat_pending_anytime) \ + { \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + pgstat_pending_anytime = true; \ + } \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); @@ -806,6 +825,8 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void); * Variables in pgstat.c */ +extern PGDLLIMPORT bool pgstat_pending_anytime; + /* GUC parameters */ extern PGDLLIMPORT bool pgstat_track_counts; extern PGDLLIMPORT int pgstat_track_functions; @@ -849,4 +870,5 @@ extern PGDLLIMPORT PgStat_Counter pgStatTransactionIdleTime; /* updated by the traffic cop and in errfinish() */ extern PGDLLIMPORT SessionEndType pgStatSessionEndCause; + #endif /* PGSTAT_H */ diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index da28afbd929..4c207611236 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -84,7 +84,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -151,7 +151,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 241945734ec..1dbc4b96f51 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --2MEBAGW8+kohXisi Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v11-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v10 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 121 +++++++++++++++--- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 22 ++++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 279 insertions(+), 66 deletions(-) 10.5% src/backend/postmaster/ 5.8% src/backend/replication/ 51.0% src/backend/utils/activity/ 5.8% src/backend/ 18.7% src/include/utils/ 6.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13cce9b49f1..cf29fc91f70 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 742137edad6..f1bae9d23d6 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -246,7 +248,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -268,6 +270,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 7c0e2809c17..bcf59227a00 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -103,7 +105,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -113,6 +115,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 7c1b8757d7d..aecc7a127e6 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index d01a09dd0c4..8c30efa2443 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3564,6 +3564,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..ddd331e2c81 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -108,10 +108,12 @@ #include "pgstat.h" #include "storage/fd.h" #include "storage/ipc.h" +#include "storage/latch.h" #include "storage/lwlock.h" #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +124,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -218,6 +219,12 @@ PgStat_LocalState pgStatLocal; */ bool pgstat_report_fixed = false; +/* + * Track when there is pending anytime flush to avoid relying on + * get_timeout_active() in hot pathes. + */ +bool pgstat_pending_anytime = false; + /* ---------- * Local data * @@ -288,6 +295,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +313,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +330,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +346,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +364,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +382,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +449,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +467,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +485,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +791,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1297,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1350,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1387,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1426,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2170,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); + + pgstat_pending_anytime = false; +} + +/* + * Timeout handler for flushing anytime stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index f2f8d3ff75f..b09316d3ab3 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 933dcb5cae5..8e86df60461 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 3277cf88a4e..6b6eec7578d 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -117,11 +117,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..2c2f3f10e10 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index b59e08605cc..eeeac1bf39a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index f16f35659b9..84e698da214 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 9bb777c3d5a..b011a315679 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -34,6 +34,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -532,8 +535,24 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !pgstat_pending_anytime) \ + { \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + pgstat_pending_anytime = true; \ + } \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); @@ -806,6 +825,8 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void); * Variables in pgstat.c */ +extern PGDLLIMPORT bool pgstat_pending_anytime; + /* GUC parameters */ extern PGDLLIMPORT bool pgstat_track_counts; extern PGDLLIMPORT int pgstat_track_functions; @@ -849,4 +870,5 @@ extern PGDLLIMPORT PgStat_Counter pgStatTransactionIdleTime; /* updated by the traffic cop and in errfinish() */ extern PGDLLIMPORT SessionEndType pgStatSessionEndCause; + #endif /* PGSTAT_H */ diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index da28afbd929..4c207611236 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -84,7 +84,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -151,7 +151,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 241945734ec..1dbc4b96f51 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --NVvBxFuyV/R+1/8/ Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v10-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v9 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 120 +++++++++++++++--- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 22 ++++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 278 insertions(+), 66 deletions(-) 10.5% src/backend/postmaster/ 5.8% src/backend/replication/ 50.9% src/backend/utils/activity/ 5.9% src/backend/ 18.8% src/include/utils/ 6.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..d01b11c7470 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 742137edad6..f1bae9d23d6 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -246,7 +248,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -268,6 +270,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 7c0e2809c17..bcf59227a00 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -103,7 +105,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -113,6 +115,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 10e64a7d1f4..11b7c114d3b 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 21de158adbb..2089de782d5 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3564,6 +3564,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..419dc512d9b 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -112,6 +112,7 @@ #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +123,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -218,6 +218,12 @@ PgStat_LocalState pgStatLocal; */ bool pgstat_report_fixed = false; +/* + * Track when there is pending anytime flush to avoid relying on + * get_timeout_active() in hot pathes. + */ +bool pgstat_pending_anytime = false; + /* ---------- * Local data * @@ -288,6 +294,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +312,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +329,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +345,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +363,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +381,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +448,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +466,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +484,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +790,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1296,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1349,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1386,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1425,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2169,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); + + pgstat_pending_anytime = false; +} + +/* + * Timeout handler for flushing anytime stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index f2f8d3ff75f..b09316d3ab3 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 933dcb5cae5..8e86df60461 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 500b1899188..c4614817966 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..2c2f3f10e10 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index b59e08605cc..eeeac1bf39a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index f16f35659b9..84e698da214 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..f0f546d419a 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,8 +536,24 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !pgstat_pending_anytime) \ + { \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + pgstat_pending_anytime = true; \ + } \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); @@ -808,6 +827,8 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void); * Variables in pgstat.c */ +extern PGDLLIMPORT bool pgstat_pending_anytime; + /* GUC parameters */ extern PGDLLIMPORT bool pgstat_track_counts; extern PGDLLIMPORT int pgstat_track_functions; @@ -851,4 +872,5 @@ extern PGDLLIMPORT PgStat_Counter pgStatTransactionIdleTime; /* updated by the traffic cop and in errfinish() */ extern PGDLLIMPORT SessionEndType pgStatSessionEndCause; + #endif /* PGSTAT_H */ diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index 64a8fe63cce..bc0b5d6e0eb 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -150,7 +150,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 241945734ec..1dbc4b96f51 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --aq/6bi8L6WORnI+9 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v9-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v8 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 112 ++++++++++++++---- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 16 +++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 264 insertions(+), 66 deletions(-) 10.8% src/backend/postmaster/ 6.0% src/backend/replication/ 50.7% src/backend/utils/activity/ 6.0% src/backend/ 19.3% src/include/utils/ 5.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..d01b11c7470 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 742137edad6..f1bae9d23d6 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -246,7 +248,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -268,6 +270,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 7c0e2809c17..bcf59227a00 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -103,7 +105,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -113,6 +115,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 10e64a7d1f4..11b7c114d3b 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 21de158adbb..2089de782d5 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3564,6 +3564,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..a4ff64dc5ce 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -112,6 +112,7 @@ #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +123,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +442,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +460,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +478,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +784,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1290,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1343,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1380,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1419,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2163,31 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} + +/* + * Timeout handler for flushing anytime stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index f2f8d3ff75f..b09316d3ab3 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 933dcb5cae5..8e86df60461 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 500b1899188..c4614817966 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..2c2f3f10e10 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index b59e08605cc..eeeac1bf39a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index f16f35659b9..84e698da214 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..b340a680614 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,8 +536,21 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index 64a8fe63cce..bc0b5d6e0eb 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -150,7 +150,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 241945734ec..1dbc4b96f51 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --OI5irqZsxWxBW9nT Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v8-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v7 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 112 ++++++++++++++---- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 16 +++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 264 insertions(+), 66 deletions(-) 10.8% src/backend/postmaster/ 6.0% src/backend/replication/ 50.7% src/backend/utils/activity/ 6.0% src/backend/ 19.3% src/include/utils/ 5.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..d01b11c7470 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 2d8f57099fd..9f8ef8159d1 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -249,7 +251,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -271,6 +273,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 23e79a32345..ded0f250288 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -106,7 +108,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -116,6 +118,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 10e64a7d1f4..11b7c114d3b 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 21de158adbb..2089de782d5 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3564,6 +3564,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..a4ff64dc5ce 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -112,6 +112,7 @@ #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +123,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +442,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +460,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +478,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +784,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1290,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1343,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1380,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1419,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2163,31 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} + +/* + * Timeout handler for flushing anytime stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index f2f8d3ff75f..b09316d3ab3 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 933dcb5cae5..8e86df60461 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 500b1899188..c4614817966 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..2c2f3f10e10 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index b59e08605cc..eeeac1bf39a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index f16f35659b9..84e698da214 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..b340a680614 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,8 +536,21 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index 64a8fe63cce..bc0b5d6e0eb 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -150,7 +150,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 241945734ec..1dbc4b96f51 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --C2xzVbxFmVrP7k6O Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v6 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 112 ++++++++++++++---- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 16 +++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 264 insertions(+), 66 deletions(-) 10.8% src/backend/postmaster/ 6.0% src/backend/replication/ 50.7% src/backend/utils/activity/ 6.0% src/backend/ 19.3% src/include/utils/ 5.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..d01b11c7470 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 2d8f57099fd..9f8ef8159d1 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -249,7 +251,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -271,6 +273,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 23e79a32345..ded0f250288 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -106,7 +108,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -116,6 +118,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 10e64a7d1f4..11b7c114d3b 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 02e9aaa6bca..c7bc409b06f 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3509,6 +3509,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..411b65aae3e 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -112,6 +112,7 @@ #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +123,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +442,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +460,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +478,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +784,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1290,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1343,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1380,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1419,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2163,31 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} + +/* + * Timeout handler for flushing non-transactional stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index 1350f5f62f1..b0bd220882d 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index d7f6d4c5ee6..61094f96a6c 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -425,7 +425,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 500b1899188..c4614817966 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..eb5f8f46925 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..f45365f47f7 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -765,6 +766,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..b340a680614 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,8 +536,21 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index 64a8fe63cce..bc0b5d6e0eb 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -150,7 +150,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 9f5ee8fd482..860f835c088 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2268,6 +2268,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --DxqFthphbM+ha9Dy Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v7 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 112 ++++++++++++++---- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 16 +++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 264 insertions(+), 66 deletions(-) 10.8% src/backend/postmaster/ 6.0% src/backend/replication/ 50.7% src/backend/utils/activity/ 6.0% src/backend/ 19.3% src/include/utils/ 5.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..d01b11c7470 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 2d8f57099fd..9f8ef8159d1 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -249,7 +251,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -271,6 +273,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 23e79a32345..ded0f250288 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -106,7 +108,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -116,6 +118,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 10e64a7d1f4..11b7c114d3b 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 21de158adbb..2089de782d5 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3564,6 +3564,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..a4ff64dc5ce 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -112,6 +112,7 @@ #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +123,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +442,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +460,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +478,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +784,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1290,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1343,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1380,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1419,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2163,31 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} + +/* + * Timeout handler for flushing anytime stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index f2f8d3ff75f..b09316d3ab3 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 933dcb5cae5..8e86df60461 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 500b1899188..c4614817966 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -116,11 +116,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..2c2f3f10e10 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index b59e08605cc..eeeac1bf39a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index f16f35659b9..84e698da214 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..b340a680614 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,8 +536,21 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index 64a8fe63cce..bc0b5d6e0eb 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -83,7 +83,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -150,7 +150,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 241945734ec..1dbc4b96f51 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --C2xzVbxFmVrP7k6O Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v11 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal and produces spikes when flushed. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - The flush_pending_cb and flush_static_cb callbacks now receive an anytime_only boolean parameter. Most of the time it's not used (except for assertions), but it's preparatory work for moving the relations stats to anytime (without introducin a new callback). - Add pgstat_schedule_anytime_update() macro to schedule the next anytime flush, relying on PGSTAT_MIN_INTERVAL The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 6 + src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/replication/walsender.c | 8 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 121 +++++++++++++++--- src/backend/utils/activity/pgstat_backend.c | 13 +- src/backend/utils/activity/pgstat_bgwriter.c | 2 +- .../utils/activity/pgstat_checkpointer.c | 2 +- src/backend/utils/activity/pgstat_database.c | 2 +- src/backend/utils/activity/pgstat_function.c | 4 +- src/backend/utils/activity/pgstat_io.c | 10 +- src/backend/utils/activity/pgstat_relation.c | 12 +- src/backend/utils/activity/pgstat_slru.c | 6 +- .../utils/activity/pgstat_subscription.c | 4 +- src/backend/utils/activity/pgstat_wal.c | 10 +- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 22 ++++ src/include/utils/pgstat_internal.h | 52 ++++++-- src/include/utils/timeout.h | 1 + .../test_custom_stats/test_custom_var_stats.c | 4 +- src/tools/pgindent/typedefs.list | 1 + 28 files changed, 279 insertions(+), 66 deletions(-) 10.5% src/backend/postmaster/ 5.8% src/backend/replication/ 51.0% src/backend/utils/activity/ 5.8% src/backend/ 18.7% src/include/utils/ 6.6% src/include/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13cce9b49f1..cf29fc91f70 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,9 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2069,9 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 0956bd39a85..059c601c3b8 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -103,7 +105,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -113,6 +115,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index e03c19123bc..e11c4b099c8 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -215,7 +216,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -225,6 +226,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index cdbe53dd262..4954fe425b7 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -245,6 +246,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 742137edad6..f1bae9d23d6 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -246,7 +248,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -268,6 +270,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 7c0e2809c17..bcf59227a00 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -103,7 +105,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); /* no query to cancel */ pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -113,6 +115,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 7c1b8757d7d..aecc7a127e6 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -252,7 +254,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -260,6 +262,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 2cde8ebc729..a7214d0dc6f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1987,8 +1987,8 @@ WalSndWaitForWal(XLogRecPtr loc) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } @@ -3016,8 +3016,8 @@ WalSndLoop(WalSndSendDataCallback send_data) if (TimestampDifferenceExceeds(last_flush, now, WALSENDER_STATS_FLUSH_INTERVAL)) { - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); last_flush = now; } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index d01a09dd0c4..8c30efa2443 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3564,6 +3564,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..ddd331e2c81 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -108,10 +108,12 @@ #include "pgstat.h" #include "storage/fd.h" #include "storage/ipc.h" +#include "storage/latch.h" #include "storage/lwlock.h" #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +124,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -218,6 +219,12 @@ PgStat_LocalState pgStatLocal; */ bool pgstat_report_fixed = false; +/* + * Track when there is pending anytime flush to avoid relying on + * get_timeout_active() in hot pathes. + */ +bool pgstat_pending_anytime = false; + /* ---------- * Local data * @@ -288,6 +295,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +313,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +330,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +346,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +364,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +382,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -436,6 +449,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +467,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +485,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +791,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,7 +1297,8 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); @@ -1345,9 +1350,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,8 +1387,22 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ - did_flush = kind_info->flush_pending_cb(entry_ref, nowait); + did_flush = kind_info->flush_pending_cb(entry_ref, nowait, anytime_only); Assert(did_flush || nowait); @@ -1402,6 +1426,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait, anytime_only); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2170,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); + + pgstat_pending_anytime = false; +} + +/* + * Timeout handler for flushing anytime stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index f2f8d3ff75f..b09316d3ab3 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,9 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +86,9 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -268,7 +275,7 @@ pgstat_flush_backend_entry_wal(PgStat_EntryRef *entry_ref) * if some statistics could not be flushed due to lock contention. */ bool -pgstat_flush_backend(bool nowait, bits32 flags) +pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only) { PgStat_EntryRef *entry_ref; bool has_pending_data = false; @@ -311,9 +318,9 @@ pgstat_flush_backend(bool nowait, bits32 flags) * If some stats could not be flushed due to lock contention, return true. */ bool -pgstat_backend_flush_cb(bool nowait) +pgstat_backend_flush_cb(bool nowait, bool anytime_only) { - return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL); + return pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_ALL, anytime_only); } /* diff --git a/src/backend/utils/activity/pgstat_bgwriter.c b/src/backend/utils/activity/pgstat_bgwriter.c index ed2fd801189..1c5f0c3ec40 100644 --- a/src/backend/utils/activity/pgstat_bgwriter.c +++ b/src/backend/utils/activity/pgstat_bgwriter.c @@ -61,7 +61,7 @@ pgstat_report_bgwriter(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_checkpointer.c b/src/backend/utils/activity/pgstat_checkpointer.c index 1f70194b7a7..2d89a082464 100644 --- a/src/backend/utils/activity/pgstat_checkpointer.c +++ b/src/backend/utils/activity/pgstat_checkpointer.c @@ -68,7 +68,7 @@ pgstat_report_checkpointer(void) /* * Report IO statistics */ - pgstat_flush_io(false); + pgstat_flush_io(false, true); } /* diff --git a/src/backend/utils/activity/pgstat_database.c b/src/backend/utils/activity/pgstat_database.c index 933dcb5cae5..8e86df60461 100644 --- a/src/backend/utils/activity/pgstat_database.c +++ b/src/backend/utils/activity/pgstat_database.c @@ -435,7 +435,7 @@ pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStatShared_Database *sharedent; PgStat_StatDBEntry *pendingent; diff --git a/src/backend/utils/activity/pgstat_function.c b/src/backend/utils/activity/pgstat_function.c index e6b84283c6c..5ba4958382f 100644 --- a/src/backend/utils/activity/pgstat_function.c +++ b/src/backend/utils/activity/pgstat_function.c @@ -190,11 +190,13 @@ pgstat_end_function_usage(PgStat_FunctionCallUsage *fcu, bool finalize) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_FunctionCounts *localent; PgStatShared_Function *shfuncent; + Assert(!anytime_only); + localent = (PgStat_FunctionCounts *) entry_ref->pending; shfuncent = (PgStatShared_Function *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..7cd32900236 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,9 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_iostats = true; pgstat_report_fixed = true; } @@ -172,9 +176,9 @@ pgstat_fetch_stat_io(void) * Simpler wrapper of pgstat_io_flush_cb() */ void -pgstat_flush_io(bool nowait) +pgstat_flush_io(bool nowait, bool anytime_only) { - (void) pgstat_io_flush_cb(nowait); + (void) pgstat_io_flush_cb(nowait, anytime_only); } /* @@ -186,7 +190,7 @@ pgstat_flush_io(bool nowait) * acquired. Otherwise, return false. */ bool -pgstat_io_flush_cb(bool nowait) +pgstat_io_flush_cb(bool nowait, bool anytime_only) { LWLock *bktype_lock; PgStat_BktypeIO *bktype_shstats; diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index bc8c43b96aa..04d21483d93 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -267,8 +267,8 @@ pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, * is done -- which will likely vacuum many relations -- or until the * VACUUM command has processed all tables and committed. */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -362,8 +362,8 @@ pgstat_report_analyze(Relation rel, pgstat_unlock_entry(entry_ref); /* see pgstat_report_vacuum() */ - pgstat_flush_io(false); - (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(false, true); + (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -812,7 +812,7 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, * entry when successfully flushing. */ bool -pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { Oid dboid; PgStat_TableStatus *lstats; /* pending stats entry */ @@ -820,6 +820,8 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) PgStat_StatTabEntry *tabentry; /* table entry of shared stats */ PgStat_StatDBEntry *dbentry; /* pending database entry */ + Assert(!anytime_only); + dboid = entry_ref->shared_entry->key.dboid; lstats = (PgStat_TableStatus *) entry_ref->pending; shtabstats = (PgStatShared_Relation *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..bf8a4d58673 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -139,7 +140,7 @@ pgstat_get_slru_index(const char *name) * acquired. Otherwise return false. */ bool -pgstat_slru_flush_cb(bool nowait) +pgstat_slru_flush_cb(bool nowait, bool anytime_only) { PgStatShared_SLRU *stats_shmem = &pgStatLocal.shmem->slru; int i; @@ -223,6 +224,9 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + pgstat_schedule_anytime_update(); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/activity/pgstat_subscription.c b/src/backend/utils/activity/pgstat_subscription.c index 3277cf88a4e..6b6eec7578d 100644 --- a/src/backend/utils/activity/pgstat_subscription.c +++ b/src/backend/utils/activity/pgstat_subscription.c @@ -117,11 +117,13 @@ pgstat_fetch_stat_subscription(Oid subid) * false without flushing the entry. Otherwise returns true. */ bool -pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) +pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_BackendSubEntry *localent; PgStatShared_Subscription *shsubent; + Assert(!anytime_only); + localent = (PgStat_BackendSubEntry *) entry_ref->pending; shsubent = (PgStatShared_Subscription *) entry_ref->shared_stats; diff --git a/src/backend/utils/activity/pgstat_wal.c b/src/backend/utils/activity/pgstat_wal.c index 183e0a7a97b..2c2f3f10e10 100644 --- a/src/backend/utils/activity/pgstat_wal.c +++ b/src/backend/utils/activity/pgstat_wal.c @@ -51,12 +51,12 @@ pgstat_report_wal(bool force) nowait = !force; /* flush wal stats */ - (void) pgstat_wal_flush_cb(nowait); - pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL); + (void) pgstat_wal_flush_cb(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_WAL, true); /* flush IO stats */ - pgstat_flush_io(nowait); - (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO); + pgstat_flush_io(nowait, true); + (void) pgstat_flush_backend(nowait, PGSTAT_BACKEND_FLUSH_IO, true); } /* @@ -88,7 +88,7 @@ pgstat_wal_have_pending(void) * acquired. Otherwise return false. */ bool -pgstat_wal_flush_cb(bool nowait) +pgstat_wal_flush_cb(bool nowait, bool anytime_only) { PgStatShared_Wal *stats_shmem = &pgStatLocal.shmem->wal; WalUsage wal_usage_diff = {0}; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index b59e08605cc..eeeac1bf39a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -773,6 +774,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index f16f35659b9..84e698da214 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 9bb777c3d5a..b011a315679 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -34,6 +34,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -532,8 +535,24 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); +/* + * Schedule the next anytime stats update timeout. + * + * This should be called whenever accumulating statistics that support + * FLUSH_ANYTIME flushing mode. + */ +#define pgstat_schedule_anytime_update() \ + do { \ + if (IsUnderPostmaster && !pgstat_pending_anytime) \ + { \ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); \ + pgstat_pending_anytime = true; \ + } \ + } while (0) + extern void pgstat_reset_counters(void); extern void pgstat_reset(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_reset_of_kind(PgStat_Kind kind); @@ -806,6 +825,8 @@ extern PgStat_WalStats *pgstat_fetch_stat_wal(void); * Variables in pgstat.c */ +extern PGDLLIMPORT bool pgstat_pending_anytime; + /* GUC parameters */ extern PGDLLIMPORT bool pgstat_track_counts; extern PGDLLIMPORT int pgstat_track_functions; @@ -849,4 +870,5 @@ extern PGDLLIMPORT PgStat_Counter pgStatTransactionIdleTime; /* updated by the traffic cop and in errfinish() */ extern PGDLLIMPORT SessionEndType pgStatSessionEndCause; + #endif /* PGSTAT_H */ diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..607f4255268 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,16 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * The mode of when to flush stats. See PgStat_FlushMode for more details. + * + * This member only has meaning for statistics kinds that accumulate + * pending stats and use flush callbacks. For kinds that write directly to + * shared memory (e.g., archiver, bgwriter, checkpointer), this member has + * no effect. + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -297,8 +320,10 @@ typedef struct PgStat_KindInfo * For variable-numbered stats: flush pending stats. Required if pending * data is used. See flush_static_cb when dealing with stats data that * that cannot use PgStat_EntryRef->pending. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait); + bool (*flush_pending_cb) (PgStat_EntryRef *sr, bool nowait, bool anytime_only); /* * For variable-numbered stats: delete pending stats. Optional. @@ -366,8 +391,10 @@ typedef struct PgStat_KindInfo * * "pgstat_report_fixed" needs to be set to trigger the flush of pending * stats. + * + * The anytime_only parameter indicates whether this is an anytime flush. */ - bool (*flush_static_cb) (bool nowait); + bool (*flush_static_cb) (bool nowait, bool anytime_only); /* * For fixed-numbered statistics: Reset All. @@ -677,6 +704,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* @@ -696,8 +724,8 @@ extern void pgstat_archiver_snapshot_cb(void); #define PGSTAT_BACKEND_FLUSH_WAL (1 << 1) /* Flush WAL statistics */ #define PGSTAT_BACKEND_FLUSH_ALL (PGSTAT_BACKEND_FLUSH_IO | PGSTAT_BACKEND_FLUSH_WAL) -extern bool pgstat_flush_backend(bool nowait, bits32 flags); -extern bool pgstat_backend_flush_cb(bool nowait); +extern bool pgstat_flush_backend(bool nowait, bits32 flags, bool anytime_only); +extern bool pgstat_backend_flush_cb(bool nowait, bool anytime_only); extern void pgstat_backend_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -729,7 +757,7 @@ extern void AtEOXact_PgStat_Database(bool isCommit, bool parallel); extern PgStat_StatDBEntry *pgstat_prep_database_pending(Oid dboid); extern void pgstat_reset_database_timestamp(Oid dboid, TimestampTz ts); -extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_database_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -737,7 +765,7 @@ extern void pgstat_database_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_function.c */ -extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_function_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -745,9 +773,9 @@ extern void pgstat_function_reset_timestamp_cb(PgStatShared_Common *header, Time * Functions in pgstat_io.c */ -extern void pgstat_flush_io(bool nowait); +extern void pgstat_flush_io(bool nowait, bool anytime_only); -extern bool pgstat_io_flush_cb(bool nowait); +extern bool pgstat_io_flush_cb(bool nowait, bool anytime_only); extern void pgstat_io_init_shmem_cb(void *stats); extern void pgstat_io_reset_all_cb(TimestampTz ts); extern void pgstat_io_snapshot_cb(void); @@ -762,7 +790,7 @@ extern void AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool extern void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); extern void PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state); -extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_relation_delete_pending_cb(PgStat_EntryRef *entry_ref); extern void pgstat_relation_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); @@ -809,7 +837,7 @@ extern PgStatShared_Common *pgstat_init_entry(PgStat_Kind kind, * Functions in pgstat_slru.c */ -extern bool pgstat_slru_flush_cb(bool nowait); +extern bool pgstat_slru_flush_cb(bool nowait, bool anytime_only); extern void pgstat_slru_init_shmem_cb(void *stats); extern void pgstat_slru_reset_all_cb(TimestampTz ts); extern void pgstat_slru_snapshot_cb(void); @@ -820,7 +848,7 @@ extern void pgstat_slru_snapshot_cb(void); */ extern void pgstat_wal_init_backend_cb(void); -extern bool pgstat_wal_flush_cb(bool nowait); +extern bool pgstat_wal_flush_cb(bool nowait, bool anytime_only); extern void pgstat_wal_init_shmem_cb(void *stats); extern void pgstat_wal_reset_all_cb(TimestampTz ts); extern void pgstat_wal_snapshot_cb(void); @@ -830,7 +858,7 @@ extern void pgstat_wal_snapshot_cb(void); * Functions in pgstat_subscription.c */ -extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait); +extern bool pgstat_subscription_flush_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only); extern void pgstat_subscription_reset_timestamp_cb(PgStatShared_Common *header, TimestampTz ts); diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/test/modules/test_custom_stats/test_custom_var_stats.c b/src/test/modules/test_custom_stats/test_custom_var_stats.c index da28afbd929..4c207611236 100644 --- a/src/test/modules/test_custom_stats/test_custom_var_stats.c +++ b/src/test/modules/test_custom_stats/test_custom_var_stats.c @@ -84,7 +84,7 @@ static dsa_area *custom_stats_description_dsa = NULL; /* Flush callback: merge pending stats into shared memory */ static bool test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, - bool nowait); + bool nowait, bool anytime_only); /* Serialization callback: write auxiliary entry data */ static void test_custom_stats_var_to_serialized_data(const PgStat_HashKey *key, @@ -151,7 +151,7 @@ _PG_init(void) * Returns false only if nowait=true and lock acquisition fails. */ static bool -test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait) +test_custom_stats_var_flush_pending_cb(PgStat_EntryRef *entry_ref, bool nowait, bool anytime_only) { PgStat_StatCustomVarEntry *pending_entry; PgStatShared_CustomVarEntry *shared_entry; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 241945734ec..1dbc4b96f51 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2271,6 +2271,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --2MEBAGW8+kohXisi Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v11-0002-Add-anytime-flush-tests-for-custom-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v1 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushBehavior enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - Register ANYTIME_STATS_UPDATE_TIMEOUT that fires every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/tcop/postgres.c | 18 +++++ src/backend/utils/activity/pgstat.c | 119 ++++++++++++++++++++++++---- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 15 ++++ src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 11 +++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 9 files changed, 154 insertions(+), 17 deletions(-) 9.7% src/backend/tcop/ 70.2% src/backend/utils/activity/ 9.3% src/backend/utils/init/ 6.0% src/include/utils/ 4.3% src/include/ diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e54bf1e760f..6a91543f80a 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,24 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + /* Skip if completely idle */ + if (!DoingCommandRead || IsTransactionOrTransactionBlock()) + pgstat_report_anytime_stat(false); + + /* Schedule next timeout */ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_ANYTIME_FLUSH_INTERVAL); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..f7942e47475 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +289,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +307,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +324,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +340,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +358,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +376,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_behavior = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +395,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +412,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +429,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +446,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +464,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +482,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +788,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1345,9 +1346,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1383,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1397,11 +1417,42 @@ pgstat_flush_pending_entries(bool nowait) cur = next; } - Assert(dlist_is_empty(&pgStatPending) == !have_pending); + /* + * When in anytime_only mode, the list may not be empty because + * FLUSH_AT_TXN_BOUNDARY entries were skipped. + */ + Assert(!anytime_only || dlist_is_empty(&pgStatPending) == !have_pending); return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2170,37 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flush non-transactional stats + * + * This is safe to call even inside a transaction. It only flushes stats + * kinds marked as FLUSH_ANYTIME. + * + * This allows long running transactions to report activity without waiting + * for transaction to finish. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* + * Exit if no pending stats at all. This avoids unnecessary work when + * backends are idle or in sessions without stats accumulation. + * + * Note: This check isn't precise as there might be only transactional + * stats pending, which we'll skip during the flush. However, maintaining + * precise tracking would add complexity that does not seem worth it from + * a performance point of view (no noticeable performance regression has + * been observed with the current implementation). + */ + if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed) + return; + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..cb0f6aecad1 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void); static void IdleSessionTimeoutHandler(void); static void IdleStatsUpdateTimeoutHandler(void); static void ClientCheckTimeoutHandler(void); +static void AnytimeStatsUpdateTimeoutHandler(void); static bool ThereIsAtLeastOneRole(void); static void process_startup_options(Port *port, bool am_superuser); static void process_settings(Oid databaseid, Oid roleid); @@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_ANYTIME_FLUSH_INTERVAL); } /* @@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void) return result; } + +/* + * Timeout handler for flushing non-transactional stats. + */ +static void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..86e65397614 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* When to call pgstat_report_anytime_stat() again */ +#define PGSTAT_ANYTIME_FLUSH_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..02f4f13fc0f 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,14 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush behavior for statistics kinds. + */ +typedef enum PgStat_FlushBehavior +{ + FLUSH_ANYTIME, /* All fields can flush anytime */ + FLUSH_AT_TXN_BOUNDARY, /* All fields need transaction boundary */ +} PgStat_FlushBehavior; /* * Metadata for a specific kind of statistics. @@ -251,6 +259,9 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* Flush behavior */ + PgStat_FlushBehavior flush_behavior; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 09e7f1d420e..9aabb325f16 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2261,6 +2261,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushBehavior PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --tjEWjIIwfNIHQLgt Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v1-0002-Remove-useless-calls-to-flush-some-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v2 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushBehavior enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - Register ANYTIME_STATS_UPDATE_TIMEOUT that fires every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/tcop/postgres.c | 16 ++++ src/backend/utils/activity/pgstat.c | 113 ++++++++++++++++++++++++---- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 15 ++++ src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 13 ++++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 9 files changed, 149 insertions(+), 16 deletions(-) 8.2% src/backend/tcop/ 69.3% src/backend/utils/activity/ 9.7% src/backend/utils/init/ 7.7% src/include/utils/ 4.5% src/include/ diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e54bf1e760f..9c4a9078ee0 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,22 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + + /* Schedule next timeout */ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_ANYTIME_FLUSH_INTERVAL); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..0f45a7d165e 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -187,7 +187,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +289,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +307,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +324,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +340,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +358,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_behavior = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +376,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_behavior = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +395,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +412,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +429,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +446,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +464,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +482,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_behavior = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +788,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1345,9 +1346,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1383,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1397,11 +1417,42 @@ pgstat_flush_pending_entries(bool nowait) cur = next; } + /* + * When in anytime_only mode, the list may not be empty because + * FLUSH_AT_TXN_BOUNDARY entries were skipped. + */ Assert(dlist_is_empty(&pgStatPending) == !have_pending); return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_behavior == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2170,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* + * Exit if no pending stats at all. This avoids unnecessary work when + * backends are idle or in sessions without stats accumulation. + * + * Note: This check isn't precise as there might be only transactional + * stats pending, which we'll skip during the flush. However, maintaining + * precise tracking would add complexity that does not seem worth it from + * a performance point of view (no noticeable performance regression has + * been observed with the current implementation). + */ + if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed) + return; + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..cb0f6aecad1 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void); static void IdleSessionTimeoutHandler(void); static void IdleStatsUpdateTimeoutHandler(void); static void ClientCheckTimeoutHandler(void); +static void AnytimeStatsUpdateTimeoutHandler(void); static bool ThereIsAtLeastOneRole(void); static void process_startup_options(Port *port, bool am_superuser); static void process_settings(Oid databaseid, Oid roleid); @@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_ANYTIME_FLUSH_INTERVAL); } /* @@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void) return result; } + +/* + * Timeout handler for flushing non-transactional stats. + */ +static void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..86e65397614 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* When to call pgstat_report_anytime_stat() again */ +#define PGSTAT_ANYTIME_FLUSH_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..63feae640d1 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,16 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush behavior for statistics kinds. + */ +typedef enum PgStat_FlushBehavior +{ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ +} PgStat_FlushBehavior; /* * Metadata for a specific kind of statistics. @@ -251,6 +261,9 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* Flush behavior */ + PgStat_FlushBehavior flush_behavior; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 3f3a888fd0e..610b35a9b31 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2268,6 +2268,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushBehavior PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --OFsrIl+bjhifp5hk Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v2-0002-Remove-useless-calls-to-flush-some-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v3 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - This relies on the existing PGSTAT_MIN_INTERVAL to fire every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/tcop/postgres.c | 16 ++++ src/backend/utils/activity/pgstat.c | 111 +++++++++++++++++++++++----- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 15 ++++ src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 16 ++++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 9 files changed, 148 insertions(+), 18 deletions(-) 8.1% src/backend/tcop/ 68.4% src/backend/utils/activity/ 9.6% src/backend/utils/init/ 9.2% src/include/utils/ 4.1% src/include/ diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e54bf1e760f..132fae61423 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,22 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + + /* Schedule next timeout */ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_MIN_INTERVAL); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..ab4d9088a9a 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -122,8 +122,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +185,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +287,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +305,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +322,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +338,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +356,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +374,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +393,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +410,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +427,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +444,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +462,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +480,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +786,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1345,9 +1344,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1381,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1402,6 +1420,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2164,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* + * Exit if no pending stats at all. This avoids unnecessary work when + * backends are idle or in sessions without stats accumulation. + * + * Note: This check isn't precise as there might be only transactional + * stats pending, which we'll skip during the flush. However, maintaining + * precise tracking would add complexity that does not seem worth it from + * a performance point of view (no noticeable performance regression has + * been observed with the current implementation). + */ + if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed) + return; + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..6076f531c4a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void); static void IdleSessionTimeoutHandler(void); static void IdleStatsUpdateTimeoutHandler(void); static void ClientCheckTimeoutHandler(void); +static void AnytimeStatsUpdateTimeoutHandler(void); static bool ThereIsAtLeastOneRole(void); static void process_startup_options(Port *port, bool am_superuser); static void process_settings(Oid databaseid, Oid roleid); @@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); } /* @@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void) return result; } + +/* + * Timeout handler for flushing non-transactional stats. + */ +static void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..1651f16f966 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..46ce90c9624 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,9 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* Flush mode */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 3f3a888fd0e..d3912b43fdc 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2268,6 +2268,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --lCc1MWfC7i2uqs8g Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v3-0002-Remove-useless-calls-to-flush-some-stats.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v4 1/4] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - This relies on the existing PGSTAT_MIN_INTERVAL to fire every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/storage/lmgr/proc.c | 10 +++ src/backend/tcop/postgres.c | 16 ++++ src/backend/utils/activity/pgstat.c | 111 +++++++++++++++++++++++----- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 15 ++++ src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 20 +++++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 10 files changed, 162 insertions(+), 18 deletions(-) 7.0% src/backend/storage/lmgr/ 7.3% src/backend/tcop/ 61.1% src/backend/utils/activity/ 8.6% src/backend/utils/init/ 11.8% src/include/utils/ 3.6% src/include/ diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c index 063826ae576..012705a2ee6 100644 --- a/src/backend/storage/lmgr/proc.c +++ b/src/backend/storage/lmgr/proc.c @@ -1322,6 +1322,7 @@ ProcSleep(LOCALLOCK *locallock) bool allow_autovacuum_cancel = true; bool logged_recovery_conflict = false; ProcWaitStatus myWaitStatus; + bool anytime_timeout_was_active = false; /* The caller must've armed the on-error cleanup mechanism */ Assert(GetAwaitedLock() == locallock); @@ -1398,6 +1399,12 @@ ProcSleep(LOCALLOCK *locallock) standbyWaitStart = GetCurrentTimestamp(); } + anytime_timeout_was_active = get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT); + + /* No need to try to flush the statistics while the process is sleeping */ + if (anytime_timeout_was_active) + disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false); + /* * If somebody wakes us between LWLockRelease and WaitLatch, the latch * will not wait. But a set latch does not necessarily mean that the lock @@ -1661,6 +1668,9 @@ ProcSleep(LOCALLOCK *locallock) } } while (myWaitStatus == PROC_WAIT_STATUS_WAITING); + if (anytime_timeout_was_active) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); + /* * Disable the timers, if they are still running. As in LockErrorCleanup, * we must preserve the LOCK_TIMEOUT indicator flag: if a lock timeout has diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e54bf1e760f..132fae61423 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,22 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + + /* Schedule next timeout */ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_MIN_INTERVAL); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..ab4d9088a9a 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -122,8 +122,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +185,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +287,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +305,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +322,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +338,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +356,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +374,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +393,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +410,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +427,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +444,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +462,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +480,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +786,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1345,9 +1344,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1381,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1402,6 +1420,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2164,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* + * Exit if no pending stats at all. This avoids unnecessary work when + * backends are idle or in sessions without stats accumulation. + * + * Note: This check isn't precise as there might be only transactional + * stats pending, which we'll skip during the flush. However, maintaining + * precise tracking would add complexity that does not seem worth it from + * a performance point of view (no noticeable performance regression has + * been observed with the current implementation). + */ + if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed) + return; + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..6076f531c4a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void); static void IdleSessionTimeoutHandler(void); static void IdleStatsUpdateTimeoutHandler(void); static void ClientCheckTimeoutHandler(void); +static void AnytimeStatsUpdateTimeoutHandler(void); static bool ThereIsAtLeastOneRole(void); static void process_startup_options(Port *port, bool am_superuser); static void process_settings(Oid databaseid, Oid roleid); @@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); } /* @@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void) return result; } + +/* + * Timeout handler for flushing non-transactional stats. + */ +static void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..1651f16f966 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..9ca39ea9a9a 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,13 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * Some stats have to be updated only at transaction boundaries (such as + * tuples_inserted updated, deleted), so it's very important to set the + * right flush mode (FLUSH_AT_TXN_BOUNDARY being the default). + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index ddbe4c64971..af21c87234a 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2268,6 +2268,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --wR/mWGXukst4NraF Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Add-GUC-to-specify-non-transactional-statistics-f.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v5 1/4] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second (if enabled while adding pending stats) to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - This relies on the existing PGSTAT_MIN_INTERVAL to fire every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/access/transam/xlog.c | 9 ++ src/backend/postmaster/bgwriter.c | 9 +- src/backend/postmaster/checkpointer.c | 10 +- src/backend/postmaster/startup.c | 2 + src/backend/postmaster/walsummarizer.c | 9 +- src/backend/postmaster/walwriter.c | 9 +- src/backend/replication/walreceiver.c | 9 +- src/backend/tcop/postgres.c | 12 ++ src/backend/utils/activity/pgstat.c | 118 ++++++++++++++++---- src/backend/utils/activity/pgstat_backend.c | 9 ++ src/backend/utils/activity/pgstat_io.c | 5 + src/backend/utils/activity/pgstat_slru.c | 5 + src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 3 + src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 21 ++++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 19 files changed, 213 insertions(+), 25 deletions(-) 5.5% src/backend/access/transam/ 15.3% src/backend/postmaster/ 3.4% src/backend/replication/ 3.8% src/backend/tcop/ 57.0% src/backend/utils/activity/ 9.2% src/include/utils/ 5.3% src/ diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..9503aea5b4d 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -1085,6 +1085,10 @@ XLogInsertRecord(XLogRecData *rdata, pgWalUsage.wal_fpi += num_fpi; pgWalUsage.wal_fpi_bytes += fpi_bytes; + /* Schedule next anytime stats update timeout */ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); + /* Required for the flush of pending stats WAL data */ pgstat_report_fixed = true; } @@ -2066,6 +2070,11 @@ AdvanceXLInsertBuffer(XLogRecPtr upto, TimeLineID tli, bool opportunistic) pgWalUsage.wal_buffers_full++; TRACE_POSTGRESQL_WAL_BUFFER_WRITE_DIRTY_DONE(); + /* Schedule next anytime stats update timeout */ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_MIN_INTERVAL); + /* * Required for the flush of pending stats WAL data, per * update of pgWalUsage. diff --git a/src/backend/postmaster/bgwriter.c b/src/backend/postmaster/bgwriter.c index 80e3088fc7e..ab5d0645026 100644 --- a/src/backend/postmaster/bgwriter.c +++ b/src/backend/postmaster/bgwriter.c @@ -49,7 +49,9 @@ #include "storage/smgr.h" #include "storage/standby.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" #include "utils/timestamp.h" /* @@ -104,7 +106,7 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -114,6 +116,11 @@ BackgroundWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * We just started, assume there has been either a shutdown or * end-of-recovery snapshot. diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index 6482c21b8f9..6e187315613 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -66,8 +66,9 @@ #include "utils/acl.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" - +#include "utils/timeout.h" /*---------- * Shared memory area for communication between checkpointer and backends @@ -216,7 +217,7 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, ReqShutdownXLOG); pqsignal(SIGTERM, SIG_IGN); /* ignore SIGTERM */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SignalHandlerForShutdownRequest); @@ -226,6 +227,11 @@ CheckpointerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Initialize so that first time-driven event happens at the correct time. */ diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index a1a4f65f9a9..498d147f0da 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -32,6 +32,7 @@ #include "storage/standby.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/timeout.h" @@ -246,6 +247,7 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len) RegisterTimeout(STANDBY_DEADLOCK_TIMEOUT, StandbyDeadLockHandler); RegisterTimeout(STANDBY_TIMEOUT, StandbyTimeoutHandler); RegisterTimeout(STANDBY_LOCK_TIMEOUT, StandbyLockTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); /* * Unblock signals (they were blocked when the postmaster forked us) diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index c3d56c866d3..cec5dfdb430 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -48,6 +48,8 @@ #include "storage/shmem.h" #include "utils/guc.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/wait_event.h" /* @@ -250,7 +252,7 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -272,6 +274,11 @@ WalSummarizerMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * If an exception is encountered, processing resumes here. */ diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 38ec8a4c8c7..7416ca703c9 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -61,7 +61,9 @@ #include "storage/smgr.h" #include "utils/hsearch.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/resowner.h" +#include "utils/timeout.h" /* @@ -107,7 +109,7 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SignalHandlerForShutdownRequest); pqsignal(SIGTERM, SignalHandlerForShutdownRequest); /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); /* not used */ @@ -117,6 +119,11 @@ WalWriterMain(const void *startup_data, size_t startup_data_len) */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* * Create a memory context that we will do all our work in. We do this so * that we can reset the context during error recovery and thereby avoid diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 8b99160ed0e..24d7ef795cb 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -77,7 +77,9 @@ #include "utils/builtins.h" #include "utils/guc.h" #include "utils/pg_lsn.h" +#include "utils/pgstat_internal.h" #include "utils/ps_status.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -253,7 +255,7 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) pqsignal(SIGINT, SIG_IGN); pqsignal(SIGTERM, die); /* request shutdown */ /* SIGQUIT handler was already set up by InitPostmasterChild */ - pqsignal(SIGALRM, SIG_IGN); + InitializeTimeouts(); /* establishes SIGALRM handler */ pqsignal(SIGPIPE, SIG_IGN); pqsignal(SIGUSR1, procsignal_sigusr1_handler); pqsignal(SIGUSR2, SIG_IGN); @@ -261,6 +263,11 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) /* Reset some signals that are accepted by postmaster but not here */ pqsignal(SIGCHLD, SIG_DFL); + /* + * Register timeouts needed + */ + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, AnytimeStatsUpdateTimeoutHandler); + /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); if (WalReceiverFunctions == NULL) diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index b4a8d2f3a1c..d19aa45400d 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,18 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..2c9454677e9 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -112,6 +112,7 @@ #include "utils/guc_hooks.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" #include "utils/timestamp.h" @@ -122,8 +123,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +186,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +288,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +306,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +323,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +339,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +357,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +375,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +394,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +411,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +428,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +445,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +463,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +481,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +787,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1293,12 +1293,18 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat if (entry_ref->pending == NULL) { - size_t entrysize = pgstat_get_kind_info(kind)->pending_size; + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + size_t entrysize = kind_info->pending_size; Assert(entrysize != (size_t) -1); entry_ref->pending = MemoryContextAllocZero(pgStatPendingContext, entrysize); dlist_push_tail(&pgStatPending, &entry_ref->pending_node); + + /* Schedule next anytime stats update timeout */ + if (kind_info->flush_mode == FLUSH_ANYTIME && IsUnderPostmaster && + !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); } return entry_ref; @@ -1345,9 +1351,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1388,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1402,6 +1427,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2171,31 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} + +/* + * Timeout handler for flushing non-transactional stats. + */ +void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/backend/utils/activity/pgstat_backend.c b/src/backend/utils/activity/pgstat_backend.c index 1350f5f62f1..9dcb24db975 100644 --- a/src/backend/utils/activity/pgstat_backend.c +++ b/src/backend/utils/activity/pgstat_backend.c @@ -31,6 +31,7 @@ #include "storage/procarray.h" #include "utils/memutils.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" /* * Backend statistics counts waiting to be flushed out. These counters may be @@ -66,6 +67,10 @@ pgstat_count_backend_io_op_time(IOObject io_object, IOContext io_context, INSTR_TIME_ADD(PendingBackendStats.pending_io.pending_times[io_object][io_context][io_op], io_time); + /* Schedule next anytime stats update timeout */ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); + backend_has_iostats = true; pgstat_report_fixed = true; } @@ -82,6 +87,10 @@ pgstat_count_backend_io_op(IOObject io_object, IOContext io_context, PendingBackendStats.pending_io.counts[io_object][io_context][io_op] += cnt; PendingBackendStats.pending_io.bytes[io_object][io_context][io_op] += bytes; + /* Schedule next anytime stats update timeout */ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); + backend_has_iostats = true; pgstat_report_fixed = true; } diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 28de24538dc..53dbf2a514b 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -19,6 +19,7 @@ #include "executor/instrument.h" #include "storage/bufmgr.h" #include "utils/pgstat_internal.h" +#include "utils/timeout.h" static PgStat_PendingIO PendingIOStats; static bool have_iostats = false; @@ -79,6 +80,10 @@ pgstat_count_io_op(IOObject io_object, IOContext io_context, IOOp io_op, /* Add the per-backend counts */ pgstat_count_backend_io_op(io_object, io_context, io_op, cnt, bytes); + /* Schedule next anytime stats update timeout */ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); + have_iostats = true; pgstat_report_fixed = true; } diff --git a/src/backend/utils/activity/pgstat_slru.c b/src/backend/utils/activity/pgstat_slru.c index 2190f388eae..1d16cde1889 100644 --- a/src/backend/utils/activity/pgstat_slru.c +++ b/src/backend/utils/activity/pgstat_slru.c @@ -19,6 +19,7 @@ #include "utils/pgstat_internal.h" #include "utils/timestamp.h" +#include "utils/timeout.h" static inline PgStat_SLRUStats *get_slru_entry(int slru_idx); @@ -223,6 +224,10 @@ get_slru_entry(int slru_idx) Assert((slru_idx >= 0) && (slru_idx < SLRU_NUM_ELEMENTS)); + /* Schedule next anytime stats update timeout */ + if (IsUnderPostmaster && !get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT)) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); + have_slrustats = true; pgstat_report_fixed = true; diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..f45365f47f7 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -64,6 +64,7 @@ #include "utils/injection_point.h" #include "utils/memutils.h" #include "utils/pg_locale.h" +#include "utils/pgstat_internal.h" #include "utils/portal.h" #include "utils/ps_status.h" #include "utils/snapmgr.h" @@ -765,6 +766,8 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); } /* diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..1651f16f966 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..a9190078d0e 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,13 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * Some stats have to be updated only at transaction boundaries (such as + * tuples_inserted updated, deleted), so it's very important to set the + * right flush mode (FLUSH_AT_TXN_BOUNDARY being the default). + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is @@ -677,6 +697,7 @@ extern PgStat_EntryRef *pgstat_fetch_pending_entry(PgStat_Kind kind, extern void *pgstat_fetch_entry(PgStat_Kind kind, Oid dboid, uint64 objid); extern void pgstat_snapshot_fixed(PgStat_Kind kind); +extern void AnytimeStatsUpdateTimeoutHandler(void); /* diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 9f5ee8fd482..860f835c088 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2268,6 +2268,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --WLtTrIUEfQDmERxy Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Add-GUC-to-specify-non-transactional-statistics-f.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
* [PATCH v4 1/4] Add pgstat_report_anytime_stat() for periodic stats flushing @ 2026-01-05 09:41 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 124+ messages in thread From: Bertrand Drouvot @ 2026-01-05 09:41 UTC (permalink / raw) Long running transactions can accumulate significant statistics (WAL, IO, ...) that remain unflushed until the transaction ends. This delays visibility of resource usage in monitoring views like pg_stat_io and pg_stat_wal. This commit introduces pgstat_report_anytime_stat(), which flushes non transactional statistics even inside active transactions. A new timeout handler fires every second to call this function, ensuring timely stats visibility without waiting for transaction completion. Implementation details: - Add PgStat_FlushMode enum to classify stats kinds: * FLUSH_ANYTIME: Stats that can always be flushed (WAL, IO, ...) * FLUSH_AT_TXN_BOUNDARY: Stats requiring transaction boundaries - Modify pgstat_flush_pending_entries() and pgstat_flush_fixed_stats() to accept a boolean anytime_only parameter: * When false: flushes all stats (existing behavior) * When true: flushes only FLUSH_ANYTIME stats and skips FLUSH_AT_TXN_BOUNDARY stats - This relies on the existing PGSTAT_MIN_INTERVAL to fire every 1 second, calling pgstat_report_anytime_stat(false) The force parameter in pgstat_report_anytime_stat() is currently unused (always called with force=false) but reserved for future use cases requiring immediate flushing. --- src/backend/storage/lmgr/proc.c | 10 +++ src/backend/tcop/postgres.c | 16 ++++ src/backend/utils/activity/pgstat.c | 111 +++++++++++++++++++++++----- src/backend/utils/init/globals.c | 1 + src/backend/utils/init/postinit.c | 15 ++++ src/include/miscadmin.h | 1 + src/include/pgstat.h | 4 + src/include/utils/pgstat_internal.h | 20 +++++ src/include/utils/timeout.h | 1 + src/tools/pgindent/typedefs.list | 1 + 10 files changed, 162 insertions(+), 18 deletions(-) 7.0% src/backend/storage/lmgr/ 7.3% src/backend/tcop/ 61.1% src/backend/utils/activity/ 8.6% src/backend/utils/init/ 11.8% src/include/utils/ 3.6% src/include/ diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c index 063826ae576..012705a2ee6 100644 --- a/src/backend/storage/lmgr/proc.c +++ b/src/backend/storage/lmgr/proc.c @@ -1322,6 +1322,7 @@ ProcSleep(LOCALLOCK *locallock) bool allow_autovacuum_cancel = true; bool logged_recovery_conflict = false; ProcWaitStatus myWaitStatus; + bool anytime_timeout_was_active = false; /* The caller must've armed the on-error cleanup mechanism */ Assert(GetAwaitedLock() == locallock); @@ -1398,6 +1399,12 @@ ProcSleep(LOCALLOCK *locallock) standbyWaitStart = GetCurrentTimestamp(); } + anytime_timeout_was_active = get_timeout_active(ANYTIME_STATS_UPDATE_TIMEOUT); + + /* No need to try to flush the statistics while the process is sleeping */ + if (anytime_timeout_was_active) + disable_timeout(ANYTIME_STATS_UPDATE_TIMEOUT, false); + /* * If somebody wakes us between LWLockRelease and WaitLatch, the latch * will not wait. But a set latch does not necessarily mean that the lock @@ -1661,6 +1668,9 @@ ProcSleep(LOCALLOCK *locallock) } } while (myWaitStatus == PROC_WAIT_STATUS_WAITING); + if (anytime_timeout_was_active) + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); + /* * Disable the timers, if they are still running. As in LockErrorCleanup, * we must preserve the LOCK_TIMEOUT indicator flag: if a lock timeout has diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index e54bf1e760f..132fae61423 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3530,6 +3530,22 @@ ProcessInterrupts(void) pgstat_report_stat(true); } + /* + * Flush stats outside of transaction boundary if the timeout fired. + * Unlike transactional stats, these can be flushed even inside a running + * transaction. + */ + if (AnytimeStatsUpdateTimeoutPending) + { + AnytimeStatsUpdateTimeoutPending = false; + + pgstat_report_anytime_stat(false); + + /* Schedule next timeout */ + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, + PGSTAT_MIN_INTERVAL); + } + if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); diff --git a/src/backend/utils/activity/pgstat.c b/src/backend/utils/activity/pgstat.c index 11bb71cad5a..ab4d9088a9a 100644 --- a/src/backend/utils/activity/pgstat.c +++ b/src/backend/utils/activity/pgstat.c @@ -122,8 +122,6 @@ * ---------- */ -/* minimum interval non-forced stats flushes.*/ -#define PGSTAT_MIN_INTERVAL 1000 /* how long until to block flushing pending stats updates */ #define PGSTAT_MAX_INTERVAL 60000 /* when to call pgstat_report_stat() again, even when idle */ @@ -187,7 +185,8 @@ static void pgstat_init_snapshot_fixed(void); static void pgstat_reset_after_failure(void); -static bool pgstat_flush_pending_entries(bool nowait); +static bool pgstat_flush_pending_entries(bool nowait, bool anytime_only); +static bool pgstat_flush_fixed_stats(bool nowait, bool anytime_only); static void pgstat_prep_snapshot(void); static void pgstat_build_snapshot(void); @@ -288,6 +287,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_database entries can be seen in all databases */ .accessed_across_databases = true, @@ -305,6 +305,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Relation), .shared_data_off = offsetof(PgStatShared_Relation, stats), @@ -321,6 +322,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .shared_size = sizeof(PgStatShared_Function), .shared_data_off = offsetof(PgStatShared_Function, stats), @@ -336,6 +338,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, .accessed_across_databases = true, @@ -353,6 +356,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = true, + .flush_mode = FLUSH_AT_TXN_BOUNDARY, /* so pg_stat_subscription_stats entries can be seen in all databases */ .accessed_across_databases = true, @@ -370,6 +374,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = false, .write_to_file = false, + .flush_mode = FLUSH_ANYTIME, .accessed_across_databases = true, @@ -388,6 +393,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, archiver), .shared_ctl_off = offsetof(PgStat_ShmemControl, archiver), @@ -404,6 +410,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, bgwriter), .shared_ctl_off = offsetof(PgStat_ShmemControl, bgwriter), @@ -420,6 +427,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, checkpointer), .shared_ctl_off = offsetof(PgStat_ShmemControl, checkpointer), @@ -436,6 +444,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, io), .shared_ctl_off = offsetof(PgStat_ShmemControl, io), @@ -453,6 +462,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, slru), .shared_ctl_off = offsetof(PgStat_ShmemControl, slru), @@ -470,6 +480,7 @@ static const PgStat_KindInfo pgstat_kind_builtin_infos[PGSTAT_KIND_BUILTIN_SIZE] .fixed_amount = true, .write_to_file = true, + .flush_mode = FLUSH_ANYTIME, .snapshot_ctl_off = offsetof(PgStat_Snapshot, wal), .shared_ctl_off = offsetof(PgStat_ShmemControl, wal), @@ -775,23 +786,11 @@ pgstat_report_stat(bool force) partial_flush = false; /* flush of variable-numbered stats tracked in pending entries list */ - partial_flush |= pgstat_flush_pending_entries(nowait); + partial_flush |= pgstat_flush_pending_entries(nowait, false); /* flush of other stats kinds */ if (pgstat_report_fixed) - { - for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) - { - const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); - - if (!kind_info) - continue; - if (!kind_info->flush_static_cb) - continue; - - partial_flush |= kind_info->flush_static_cb(nowait); - } - } + partial_flush |= pgstat_flush_fixed_stats(nowait, false); last_flush = now; @@ -1345,9 +1344,14 @@ pgstat_delete_pending_entry(PgStat_EntryRef *entry_ref) /* * Flush out pending variable-numbered stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME entries. + * This is safe to call inside transactions. + * + * If anytime_only is false, flushes all entries. */ static bool -pgstat_flush_pending_entries(bool nowait) +pgstat_flush_pending_entries(bool nowait, bool anytime_only) { bool have_pending = false; dlist_node *cur = NULL; @@ -1377,6 +1381,20 @@ pgstat_flush_pending_entries(bool nowait) Assert(!kind_info->fixed_amount); Assert(kind_info->flush_pending_cb != NULL); + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + { + have_pending = true; + + if (dlist_has_next(&pgStatPending, cur)) + next = dlist_next_node(&pgStatPending, cur); + else + next = NULL; + + cur = next; + continue; + } + /* flush the stats, if possible */ did_flush = kind_info->flush_pending_cb(entry_ref, nowait); @@ -1402,6 +1420,33 @@ pgstat_flush_pending_entries(bool nowait) return have_pending; } +/* + * Flush fixed-amount stats. + * + * If anytime_only is true, only flushes FLUSH_ANYTIME stats (safe inside transactions). + * If anytime_only is false, flushes all stats with flush_static_cb. + */ +static bool +pgstat_flush_fixed_stats(bool nowait, bool anytime_only) +{ + bool partial_flush = false; + + for (PgStat_Kind kind = PGSTAT_KIND_MIN; kind <= PGSTAT_KIND_MAX; kind++) + { + const PgStat_KindInfo *kind_info = pgstat_get_kind_info(kind); + + if (!kind_info || !kind_info->flush_static_cb) + continue; + + /* Skip transactional stats if we're in anytime_only mode */ + if (anytime_only && kind_info->flush_mode == FLUSH_AT_TXN_BOUNDARY) + continue; + + partial_flush |= kind_info->flush_static_cb(nowait); + } + + return partial_flush; +} /* ------------------------------------------------------------ * Helper / infrastructure functions @@ -2119,3 +2164,33 @@ assign_stats_fetch_consistency(int newval, void *extra) if (pgstat_fetch_consistency != newval) force_stats_snapshot_clear = true; } + +/* + * Flushes only FLUSH_ANYTIME stats using non-blocking locks. Transactional + * stats (FLUSH_AT_TXN_BOUNDARY) remain pending until transaction boundary. + * Safe to call inside transactions. + */ +void +pgstat_report_anytime_stat(bool force) +{ + bool nowait = !force; + + pgstat_assert_is_up(); + + /* + * Exit if no pending stats at all. This avoids unnecessary work when + * backends are idle or in sessions without stats accumulation. + * + * Note: This check isn't precise as there might be only transactional + * stats pending, which we'll skip during the flush. However, maintaining + * precise tracking would add complexity that does not seem worth it from + * a performance point of view (no noticeable performance regression has + * been observed with the current implementation). + */ + if (dlist_is_empty(&pgStatPending) && !pgstat_report_fixed) + return; + + /* Flush stats outside of transaction boundary */ + pgstat_flush_pending_entries(nowait, true); + pgstat_flush_fixed_stats(nowait, true); +} diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c index 36ad708b360..ad44826c39e 100644 --- a/src/backend/utils/init/globals.c +++ b/src/backend/utils/init/globals.c @@ -40,6 +40,7 @@ volatile sig_atomic_t IdleSessionTimeoutPending = false; volatile sig_atomic_t ProcSignalBarrierPending = false; volatile sig_atomic_t LogMemoryContextPending = false; volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; +volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending = false; volatile uint32 InterruptHoldoffCount = 0; volatile uint32 QueryCancelHoldoffCount = 0; volatile uint32 CritSectionCount = 0; diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 3f401faf3de..6076f531c4a 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -82,6 +82,7 @@ static void TransactionTimeoutHandler(void); static void IdleSessionTimeoutHandler(void); static void IdleStatsUpdateTimeoutHandler(void); static void ClientCheckTimeoutHandler(void); +static void AnytimeStatsUpdateTimeoutHandler(void); static bool ThereIsAtLeastOneRole(void); static void process_startup_options(Port *port, bool am_superuser); static void process_settings(Oid databaseid, Oid roleid); @@ -765,6 +766,9 @@ InitPostgres(const char *in_dbname, Oid dboid, RegisterTimeout(CLIENT_CONNECTION_CHECK_TIMEOUT, ClientCheckTimeoutHandler); RegisterTimeout(IDLE_STATS_UPDATE_TIMEOUT, IdleStatsUpdateTimeoutHandler); + RegisterTimeout(ANYTIME_STATS_UPDATE_TIMEOUT, + AnytimeStatsUpdateTimeoutHandler); + enable_timeout_after(ANYTIME_STATS_UPDATE_TIMEOUT, PGSTAT_MIN_INTERVAL); } /* @@ -1446,3 +1450,14 @@ ThereIsAtLeastOneRole(void) return result; } + +/* + * Timeout handler for flushing non-transactional stats. + */ +static void +AnytimeStatsUpdateTimeoutHandler(void) +{ + AnytimeStatsUpdateTimeoutPending = true; + InterruptPending = true; + SetLatch(MyLatch); +} diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index db559b39c4d..8aeb9628871 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending; extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending; extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending; +extern PGDLLIMPORT volatile sig_atomic_t AnytimeStatsUpdateTimeoutPending; extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending; extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost; diff --git a/src/include/pgstat.h b/src/include/pgstat.h index fff7ecc2533..1651f16f966 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -35,6 +35,9 @@ /* Default directory to store temporary statistics data in */ #define PG_STAT_TMP_DIR "pg_stat_tmp" +/* Minimum interval non-forced stats flushes */ +#define PGSTAT_MIN_INTERVAL 1000 + /* Values for track_functions GUC variable --- order is significant! */ typedef enum TrackFunctionsLevel { @@ -533,6 +536,7 @@ extern void pgstat_initialize(void); /* Functions called from backends */ extern long pgstat_report_stat(bool force); +extern void pgstat_report_anytime_stat(bool force); extern void pgstat_force_next_flush(void); extern void pgstat_reset_counters(void); diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h index 9b8fbae00ed..9ca39ea9a9a 100644 --- a/src/include/utils/pgstat_internal.h +++ b/src/include/utils/pgstat_internal.h @@ -224,6 +224,19 @@ typedef struct PgStat_SubXactStatus PgStat_TableXactStatus *first; /* head of list for this subxact */ } PgStat_SubXactStatus; +/* + * Flush mode for statistics kinds. + * + * FLUSH_AT_TXN_BOUNDARY has to be the first because we want it to be the + * default value. + */ +typedef enum PgStat_FlushMode +{ + FLUSH_AT_TXN_BOUNDARY, /* All fields can only be flushed at + * transaction boundary */ + FLUSH_ANYTIME, /* All fields can be flushed anytime, + * including within transactions */ +} PgStat_FlushMode; /* * Metadata for a specific kind of statistics. @@ -251,6 +264,13 @@ typedef struct PgStat_KindInfo */ bool track_entry_count:1; + /* + * Some stats have to be updated only at transaction boundaries (such as + * tuples_inserted updated, deleted), so it's very important to set the + * right flush mode (FLUSH_AT_TXN_BOUNDARY being the default). + */ + PgStat_FlushMode flush_mode; + /* * The size of an entry in the shared stats hash table (pointed to by * PgStatShared_HashEntry->body). For fixed-numbered statistics, this is diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h index 0965b590b34..10723bb664c 100644 --- a/src/include/utils/timeout.h +++ b/src/include/utils/timeout.h @@ -35,6 +35,7 @@ typedef enum TimeoutId IDLE_SESSION_TIMEOUT, IDLE_STATS_UPDATE_TIMEOUT, CLIENT_CONNECTION_CHECK_TIMEOUT, + ANYTIME_STATS_UPDATE_TIMEOUT, STARTUP_PROGRESS_TIMEOUT, /* First user-definable timeout reason */ USER_TIMEOUT, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index ddbe4c64971..af21c87234a 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2268,6 +2268,7 @@ PgStat_Counter PgStat_EntryRef PgStat_EntryRefHashEntry PgStat_FetchConsistency +PgStat_FlushMode PgStat_FunctionCallUsage PgStat_FunctionCounts PgStat_HashKey -- 2.34.1 --wR/mWGXukst4NraF Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Add-GUC-to-specify-non-transactional-statistics-f.patch" ^ permalink raw reply [nested|flat] 124+ messages in thread
end of thread, other threads:[~2026-01-05 09:41 UTC | newest] Thread overview: 124+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2019-07-01 15:11 [PATCH v1] Fix usage of unified logging pg_log_* in pg_rewind and initdb Alexey Kondratov <[email protected]> 2023-05-16 18:11 Re: Schema variables - new implementation for Postgres 15 Pavel Stehule <[email protected]> 2023-05-17 03:20 ` Re: Schema variables - new implementation for Postgres 15 Pavel Stehule <[email protected]> 2023-06-25 11:48 [PATCH v1 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-06-25 11:48 [PATCH v1 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-06-25 11:48 [PATCH v1 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-06-25 11:48 [PATCH v1 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-06-25 11:48 [PATCH v1 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-06-26 08:05 [PATCH v2 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-06-26 08:05 [PATCH v2 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-06-26 08:05 [PATCH v2 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-06-26 08:05 [PATCH v2 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-06-26 08:05 [PATCH v2 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-07-26 10:49 [PATCH v3 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-07-26 10:49 [PATCH v3 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-07-26 10:49 [PATCH v3 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-07-26 10:49 [PATCH v3 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-08-09 07:56 [PATCH v4 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-08-09 07:56 [PATCH v4 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-08-09 07:56 [PATCH v4 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-08-09 07:56 [PATCH v4 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-08-09 07:56 [PATCH v4 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-08-09 07:56 [PATCH v4 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-02 06:32 [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-02 06:32 [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-02 06:32 [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-02 06:32 [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-02 06:32 [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-02 06:32 [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-02 06:32 [PATCH v5 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-12 05:22 [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-12 05:22 [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-12 05:22 [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-12 05:22 [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-12 05:22 [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-12 05:22 [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-12 05:22 [PATCH v6 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-22 04:53 [PATCH v7 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-22 04:53 [PATCH v7 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-22 04:53 [PATCH v7 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-22 04:53 [PATCH v7 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-22 04:53 [PATCH v7 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-25 05:01 [PATCH v8 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-25 05:01 [PATCH v8 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-25 05:01 [PATCH v8 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-09-25 05:01 [PATCH v8 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-10-04 05:51 [PATCH v9 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-10-04 05:51 [PATCH v9 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-10-04 05:51 [PATCH v9 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-10-04 05:51 [PATCH v9 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-10-22 02:22 [PATCH v10 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-10-22 02:22 [PATCH v10 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-10-22 02:22 [PATCH v10 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-10-22 02:22 [PATCH v10 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-11-08 06:57 [PATCH v11 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-11-08 06:57 [PATCH v11 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-11-08 06:57 [PATCH v11 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-12-04 11:23 [PATCH v12 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-12-04 11:23 [PATCH v12 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2023-12-04 11:23 [PATCH v12 2/7] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-01-22 09:45 [PATCH v13 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-01-22 09:45 [PATCH v13 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-01-22 09:45 [PATCH v13 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-02-28 13:59 [PATCH v14 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-02-28 13:59 [PATCH v14 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-02-28 13:59 [PATCH v14 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-03-28 10:30 [PATCH v15 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-03-28 10:30 [PATCH v15 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-03-28 10:30 [PATCH v15 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-03-28 10:30 [PATCH v15 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-03-28 10:30 [PATCH v15 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-04-12 06:49 [PATCH v16 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-04-12 06:49 [PATCH v16 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-04-12 06:49 [PATCH v16 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-04-28 11:00 [PATCH v17 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-04-28 11:00 [PATCH v17 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-04-28 11:00 [PATCH v17 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-05-11 07:11 [PATCH v18 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-05-11 07:11 [PATCH v18 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-05-11 07:11 [PATCH v18 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-05-14 23:26 [PATCH v19 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-05-14 23:26 [PATCH v19 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-05-14 23:26 [PATCH v19 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-05-24 02:26 [PATCH v20 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-05-24 02:26 [PATCH v20 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-05-24 02:26 [PATCH v20 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-08-26 04:32 [PATCH v21 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-08-26 04:32 [PATCH v21 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-08-26 04:32 [PATCH v21 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-09-19 04:48 [PATCH v22 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-09-19 04:48 [PATCH v22 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-09-19 04:48 [PATCH v22 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-10-25 03:56 [PATCH v23 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-10-25 03:56 [PATCH v23 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-10-25 03:56 [PATCH v23 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-12-19 06:06 [PATCH v24 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-12-19 06:06 [PATCH v24 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-12-19 06:06 [PATCH v24 2/8] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-12-21 06:19 [PATCH v25 2/9] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-12-21 06:19 [PATCH v25 2/9] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-12-21 06:19 [PATCH v25 2/9] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-12-30 12:44 [PATCH v26 2/9] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-12-30 12:44 [PATCH v26 2/9] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-12-30 12:44 [PATCH v26 2/9] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-12-30 23:53 [PATCH v27 2/9] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-12-30 23:53 [PATCH v27 2/9] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2024-12-30 23:53 [PATCH v27 2/9] Row pattern recognition patch (parse/analysis). Tatsuo Ishii <[email protected]> 2026-01-05 09:41 [PATCH v9 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v10 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v3 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v4 1/4] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v7 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v8 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v5 1/4] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v3 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v2 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v6 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v1 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v4 1/4] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v7 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v11 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v2 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v11 1/5] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]> 2026-01-05 09:41 [PATCH v1 1/3] Add pgstat_report_anytime_stat() for periodic stats flushing Bertrand Drouvot <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox