public inbox for [email protected]
help / color / mirror / Atom feedRe: Cached plans and statement generalization
51+ messages / 11 participants
[nested] [flat]
* Re: Cached plans and statement generalization
@ 2017-05-01 15:52 Robert Haas <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Robert Haas @ 2017-05-01 15:52 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On Fri, Apr 28, 2017 at 6:01 AM, Konstantin Knizhnik <
[email protected]> wrote:
>
> Any comments and suggestions for future improvement of this patch are
> welcome.
>
+ PG_TRY();
+ {
+ query = parse_analyze_varparams(parse_tree,
+ query_string,
+ ¶m_types,
+ &num_params);
+ }
+ PG_CATCH();
+ {
+ /*
+ * In case of analyze errors revert back to original query
processing
+ * and disable autoprepare for this query to avoid such
problems in future.
+ */
+ FlushErrorState();
+ if (snapshot_set) {
+ PopActiveSnapshot();
+ }
+ entry->disable_autoprepare = true;
+ undo_query_plan_changes(parse_tree, const_param_list);
+ MemoryContextSwitchTo(old_context);
+ return false;
+ }
+ PG_END_TRY();
This is definitely not a safe way of using TRY/CATCH.
+
+ /* Convert literal value to parameter value */
+ switch (const_param->literal->val.type)
+ {
+ /*
+ * Convert from integer literal
+ */
+ case T_Integer:
+ switch (ptype) {
+ case INT8OID:
+ params->params[paramno].value =
Int64GetDatum((int64)const_param->literal->val.val.ival);
+ break;
+ case INT4OID:
+ params->params[paramno].value =
Int32GetDatum((int32)const_param->literal->val.val.ival);
+ break;
+ case INT2OID:
+ if (const_param->literal->val.val.ival < SHRT_MIN
+ || const_param->literal->val.val.ival > SHRT_MAX)
+ {
+ ereport(ERROR,
+
(errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg("smallint out of range")));
+ }
+ params->params[paramno].value =
Int16GetDatum((int16)const_param->literal->val.val.ival);
+ break;
+ case FLOAT4OID:
+ params->params[paramno].value =
Float4GetDatum((float)const_param->literal->val.val.ival);
+ break;
+ case FLOAT8OID:
+ params->params[paramno].value =
Float8GetDatum((double)const_param->literal->val.val.ival);
+ break;
+ case INT4RANGEOID:
+ sprintf(buf, "[%ld,%ld]",
const_param->literal->val.val.ival, const_param->literal->val.val.ival);
+ getTypeInputInfo(ptype, &typinput, &typioparam);
+ params->params[paramno].value =
OidInputFunctionCall(typinput, buf, typioparam, -1);
+ break;
+ default:
+ pg_lltoa(const_param->literal->val.val.ival, buf);
+ getTypeInputInfo(ptype, &typinput, &typioparam);
+ params->params[paramno].value =
OidInputFunctionCall(typinput, buf, typioparam, -1);
+ }
+ break;
+ case T_Null:
+ params->params[paramno].isnull = true;
+ break;
+ default:
+ /*
+ * Convert from string literal
+ */
+ getTypeInputInfo(ptype, &typinput, &typioparam);
+ params->params[paramno].value =
OidInputFunctionCall(typinput, const_param->literal->val.val.str,
typioparam, -1);
+ }
I don't see something with a bunch of hard-coded rules for particular type
OIDs having any chance of being acceptable.
This patch seems to duplicate a large amount of existing code. That would
be a good thing to avoid.
It could use a visit from the style police and a spell-checker, too.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-02 09:50 Konstantin Knizhnik <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Konstantin Knizhnik @ 2017-05-02 09:50 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On 01.05.2017 18:52, Robert Haas wrote:
> On Fri, Apr 28, 2017 at 6:01 AM, Konstantin Knizhnik
> <[email protected] <mailto:[email protected]>> wrote:
>
>
> Any comments and suggestions for future improvement of this patch
> are welcome.
>
>
> + PG_TRY();
> + {
> + query = parse_analyze_varparams(parse_tree,
> + query_string,
> + ¶m_types,
> + &num_params);
> + }
> + PG_CATCH();
> + {
> + /*
> + * In case of analyze errors revert back to original
> query processing
> + * and disable autoprepare for this query to avoid such
> problems in future.
> + */
> + FlushErrorState();
> + if (snapshot_set) {
> + PopActiveSnapshot();
> + }
> + entry->disable_autoprepare = true;
> + undo_query_plan_changes(parse_tree, const_param_list);
> + MemoryContextSwitchTo(old_context);
> + return false;
> + }
> + PG_END_TRY();
>
> This is definitely not a safe way of using TRY/CATCH.
>
> +
> + /* Convert literal value to parameter value */
> + switch (const_param->literal->val.type)
> + {
> + /*
> + * Convert from integer literal
> + */
> + case T_Integer:
> + switch (ptype) {
> + case INT8OID:
> + params->params[paramno].value =
> Int64GetDatum((int64)const_param->literal->val.val.ival);
> + break;
> + case INT4OID:
> + params->params[paramno].value =
> Int32GetDatum((int32)const_param->literal->val.val.ival);
> + break;
> + case INT2OID:
> + if (const_param->literal->val.val.ival < SHRT_MIN
> + || const_param->literal->val.val.ival > SHRT_MAX)
> + {
> + ereport(ERROR,
> + (errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
> + errmsg("smallint out of range")));
> + }
> + params->params[paramno].value =
> Int16GetDatum((int16)const_param->literal->val.val.ival);
> + break;
> + case FLOAT4OID:
> + params->params[paramno].value =
> Float4GetDatum((float)const_param->literal->val.val.ival);
> + break;
> + case FLOAT8OID:
> + params->params[paramno].value =
> Float8GetDatum((double)const_param->literal->val.val.ival);
> + break;
> + case INT4RANGEOID:
> + sprintf(buf, "[%ld,%ld]",
> const_param->literal->val.val.ival, const_param->literal->val.val.ival);
> + getTypeInputInfo(ptype, &typinput, &typioparam);
> + params->params[paramno].value =
> OidInputFunctionCall(typinput, buf, typioparam, -1);
> + break;
> + default:
> + pg_lltoa(const_param->literal->val.val.ival, buf);
> + getTypeInputInfo(ptype, &typinput, &typioparam);
> + params->params[paramno].value =
> OidInputFunctionCall(typinput, buf, typioparam, -1);
> + }
> + break;
> + case T_Null:
> + params->params[paramno].isnull = true;
> + break;
> + default:
> + /*
> + * Convert from string literal
> + */
> + getTypeInputInfo(ptype, &typinput, &typioparam);
> + params->params[paramno].value =
> OidInputFunctionCall(typinput, const_param->literal->val.val.str,
> typioparam, -1);
> + }
>
> I don't see something with a bunch of hard-coded rules for particular
> type OIDs having any chance of being acceptable.
>
Well, what I need is to convert literal value represented in Value
struct to parameter datum value.
Struct "value" contains union with integer literal and text.
So this peace of code is just provides efficient handling of most common
cases (integer parameters) and uses type's input function in other cases.
> This patch seems to duplicate a large amount of existing code. That
> would be a good thing to avoid.
Yes, I have to copy a lot of code from exec_parse_message +
exec_bind_message + exec_execute_message functions.
Definitely copying of code is bad flaw. It will be much better and
easier just to call three original functions instead of mixing gathering
their code into the new function.
But I failed to do it because
1. Autoprepare should be integrated into exec_simple_query. Before
executing query in normal way, I need to perform cache lookup for
previously prepared plan for this generalized query.
And generalization of query requires building of query tree (query
parsing). In other words, parsing should be done before I can call
exec_parse_message.
2. exec_bind_message works with parameters passed by client though
libpwq protocol, while autoprepare deals with values of parameters
extracted from literals.
3. I do not want to generate dummy name for autoprepared query to handle
it as normal prepared statement.
And I can not use unnamed statements because I want lifetime of
autoprepared statements will be larger than one statement.
4. I have to use slightly different memory context policy than named or
unnamed prepared statements.
Also this three exec_* functions contain prolog/epilog code which is
needed because them are serving separate libpq requests.
But in case of autoprepared statements them need to be executed in the
context of single libpq message, so most of this code is redundant.
> It could use a visit from the style police and a spell-checker, too.
I will definitely fix style and misspelling - I have not submitted yet
this patch to commit fest and there is long enough time to next commitfest.
My primary intention of publishing this patch is receive feedback on the
proposed approach.
I already got two very useful advices: limit number of cached statements
and pay more attention to safety.
This is why I have reimplemented my original approach with substituting
string literals with parameters without building parse tree.
Right now I am mostly thinking about two problems:
1. Finding out best criteria of detecting literals which need to be
replaced with parameters and which not. It is clear that replacing
"limit 10" with "limit $10"
will have not so much sense and can cause worse execution plan. So right
now I just ignore sort, group by and limit parts. But may be it is
possible to find some more flexible approach.
2. Which type to chose for parameters. I can try to infer type from
context (current solution), or try to use type of literal.
The problem with first approach is that query compiler is not always
able to do it and even if type can be determined, it may be too generic
(for example numeric instead of real
or range instead of integer). The problem with second approach is
opposite: type inferred from literal type can be too restrictive - quite
often integer literals are used to specify values of floating point
constant. The best solution is first try to determine parameter type
from context and then refine it based on literal type. But it will
require repeat of query analysis.
Not sure if it is possible.
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
Thanks for your feedback.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-02 18:26 Robert Haas <[email protected]>
parent: Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Robert Haas @ 2017-05-02 18:26 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On Tue, May 2, 2017 at 5:50 AM, Konstantin Knizhnik
<[email protected]> wrote:
>> I don't see something with a bunch of hard-coded rules for particular type
>> OIDs having any chance of being acceptable.
>
> Well, what I need is ...
Regarding this...
> Definitely copying of code is bad flaw. It will be much better and easier
> just to call three original functions instead of mixing gathering their code
> into the new function.
> But I failed to do it because ...
...and also this:
I am sympathetic to the fact that this is a hard problem to solve.
I'm just telling you that the way you've got it is not acceptable and
nobody's going to commit it like that (or if they do, they will end up
having to revert it). If you want to have a technical discussion
about what might be a way to change the patch to be more acceptable,
cool, but I don't want to get into a long debate about whether what
you have is acceptable or not; I've already said what I think about
that and I believe that opinion will be widely shared. I am not
trying to beat you up here, just trying to be clear.
> Thanks for your feedback.
Sure thing.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-10 16:11 Konstantin Knizhnik <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 3 replies; 51+ messages in thread
From: Konstantin Knizhnik @ 2017-05-10 16:11 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On 02.05.2017 21:26, Robert Haas wrote:
>
> I am sympathetic to the fact that this is a hard problem to solve.
> I'm just telling you that the way you've got it is not acceptable and
> nobody's going to commit it like that (or if they do, they will end up
> having to revert it). If you want to have a technical discussion
> about what might be a way to change the patch to be more acceptable,
> cool, but I don't want to get into a long debate about whether what
> you have is acceptable or not; I've already said what I think about
> that and I believe that opinion will be widely shared. I am not
> trying to beat you up here, just trying to be clear.
>
>
Based on the Robert's feedback and Tom's proposal I have implemented two
new versions of autoprepare patch.
First version is just refactoring of my original implementation: I have
extracted common code into prepare_cached_plan and exec_prepared_plan
function to avoid code duplication. Also I rewrote assignment of values
to parameters. Now types of parameters are inferred from types of
literals, so there may be several
prepared plans which are different only by types of parameters. Due to
the problem with type coercion for parameters, I have to catch errors in
parse_analyze_varparams.
Robert, can you please explain why using TRY/CATCH is not safe here:
> This is definitely not a safe way of using TRY/CATCH.
Second version is based on Tom's suggestion:
> Personally I'd think about
> replacing the entire literal-with-cast construct with a Param having
> already-known type.
So here I first patch raw parse tree, replacing A_Const with ParamRef.
Such plan is needed to perform cache lookup.
Then I restore original raw parse tree and call pg_analyze_and_rewrite.
Then I mutate analyzed tree, replacing Const with Param nodes.
In this case type coercion is already performed and I know precise types
which should be used for parameters.
It seems to be more sophisticated approach. And I can not extract common
code in prepare_cached_plan,
because preparing of plan is currently mix of steps done in
exec_simple_query and exec_parse_message.
But there is no need to catch analyze errors.
Finally performance of both approaches is the same: at pgbench it is
180k TPS on read-only queries, comparing with 80k TPS for not prepared
queries.
In both cases 7 out of 177 regression tests are not passed (mostly
because session environment is changed between subsequent query execution).
I am going to continue work on this patch I will be glad to receive any
feedback and suggestions for its improvement.
In most cases, applications are not accessing Postgres directly, but
using some connection pooling layer and so them are not able to use
prepared statements.
But at simple OLTP Postgres spent more time on building query plan than
on execution itself. And it is possible to speedup Postgres about two
times at such workload!
Another alternative is true shared plan cache. May be it is even more
perspective approach, but definitely much more invasive and harder to
implement.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-patch] autoprepare-1.patch (46.7K, ../../[email protected]/2-autoprepare-1.patch)
download | inline diff:
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index 6e52eb7..f2eb0f5 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -3696,6 +3696,454 @@ raw_expression_tree_walker(Node *node,
}
/*
+ * raw_expression_tree_mutator --- transform raw parse tree.
+ *
+ * This function is implementing slightly different approach for tree update than expression_tree_mutator().
+ * Callback is given pointer to pointer to the current node and can update this field instead of returning reference to new node.
+ * It makes it possible to remember changes and easily revert them without extra traversal of the tree.
+ *
+ * This function do not need QTW_DONT_COPY_QUERY flag: it never implicitly copy tree nodes, doing in-place update.
+ *
+ * Like raw_expression_tree_walker, there is no special rule about query
+ * boundaries: we descend to everything that's possibly interesting.
+ *
+ * Currently, the node type coverage here extends only to DML statements
+ * (SELECT/INSERT/UPDATE/DELETE) and nodes that can appear in them, because
+ * this is used mainly during analysis of CTEs, and only DML statements can
+ * appear in CTEs. If some other node is visited, iteration is immediately stopped and true is returned.
+ */
+bool
+raw_expression_tree_mutator(Node *node,
+ bool (*mutator) (),
+ void *context)
+{
+ ListCell *temp;
+
+ /*
+ * The walker has already visited the current node, and so we need only
+ * recurse into any sub-nodes it has.
+ */
+ if (node == NULL)
+ return false;
+
+ /* Guard against stack overflow due to overly complex expressions */
+ check_stack_depth();
+
+ switch (nodeTag(node))
+ {
+ case T_SetToDefault:
+ case T_CurrentOfExpr:
+ case T_Integer:
+ case T_Float:
+ case T_String:
+ case T_BitString:
+ case T_Null:
+ case T_ParamRef:
+ case T_A_Const:
+ case T_A_Star:
+ /* primitive node types with no subnodes */
+ break;
+ case T_Alias:
+ /* we assume the colnames list isn't interesting */
+ break;
+ case T_RangeVar:
+ return mutator(&((RangeVar *) node)->alias, context);
+ case T_GroupingFunc:
+ return mutator(&((GroupingFunc *) node)->args, context);
+ case T_SubLink:
+ {
+ SubLink *sublink = (SubLink *) node;
+
+ if (mutator(&sublink->testexpr, context))
+ return true;
+ /* we assume the operName is not interesting */
+ if (mutator(&sublink->subselect, context))
+ return true;
+ }
+ break;
+ case T_CaseExpr:
+ {
+ CaseExpr *caseexpr = (CaseExpr *) node;
+
+ if (mutator(&caseexpr->arg, context))
+ return true;
+ /* we assume mutator(& doesn't care about CaseWhens, either */
+ foreach(temp, caseexpr->args)
+ {
+ CaseWhen *when = (CaseWhen *) lfirst(temp);
+
+ Assert(IsA(when, CaseWhen));
+ if (mutator(&when->expr, context))
+ return true;
+ if (mutator(&when->result, context))
+ return true;
+ }
+ if (mutator(&caseexpr->defresult, context))
+ return true;
+ }
+ break;
+ case T_RowExpr:
+ /* Assume colnames isn't interesting */
+ return mutator(&((RowExpr *) node)->args, context);
+ case T_CoalesceExpr:
+ return mutator(&((CoalesceExpr *) node)->args, context);
+ case T_MinMaxExpr:
+ return mutator(&((MinMaxExpr *) node)->args, context);
+ case T_XmlExpr:
+ {
+ XmlExpr *xexpr = (XmlExpr *) node;
+
+ if (mutator(&xexpr->named_args, context))
+ return true;
+ /* we assume mutator(& doesn't care about arg_names */
+ if (mutator(&xexpr->args, context))
+ return true;
+ }
+ break;
+ case T_NullTest:
+ return mutator(&((NullTest *) node)->arg, context);
+ case T_BooleanTest:
+ return mutator(&((BooleanTest *) node)->arg, context);
+ case T_JoinExpr:
+ {
+ JoinExpr *join = (JoinExpr *) node;
+
+ if (mutator(&join->larg, context))
+ return true;
+ if (mutator(&join->rarg, context))
+ return true;
+ if (mutator(&join->quals, context))
+ return true;
+ if (mutator(&join->alias, context))
+ return true;
+ /* using list is deemed uninteresting */
+ }
+ break;
+ case T_IntoClause:
+ {
+ IntoClause *into = (IntoClause *) node;
+
+ if (mutator(&into->rel, context))
+ return true;
+ /* colNames, options are deemed uninteresting */
+ /* viewQuery should be null in raw parsetree, but check it */
+ if (mutator(&into->viewQuery, context))
+ return true;
+ }
+ break;
+ case T_List:
+ foreach(temp, (List *) node)
+ {
+ if (mutator(&lfirst(temp), context))
+ return true;
+ }
+ break;
+ case T_InsertStmt:
+ {
+ InsertStmt *stmt = (InsertStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->cols, context))
+ return true;
+ if (mutator(&stmt->selectStmt, context))
+ return true;
+ if (mutator(&stmt->onConflictClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_DeleteStmt:
+ {
+ DeleteStmt *stmt = (DeleteStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->usingClause, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_UpdateStmt:
+ {
+ UpdateStmt *stmt = (UpdateStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->fromClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_SelectStmt:
+ {
+ SelectStmt *stmt = (SelectStmt *) node;
+
+ if (mutator(&stmt->distinctClause, context))
+ return true;
+ if (mutator(&stmt->intoClause, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->fromClause, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->groupClause, context))
+ return true;
+ if (mutator(&stmt->havingClause, context))
+ return true;
+ if (mutator(&stmt->windowClause, context))
+ return true;
+ if (mutator(&stmt->valuesLists, context))
+ return true;
+ if (mutator(&stmt->sortClause, context))
+ return true;
+ if (mutator(&stmt->limitOffset, context))
+ return true;
+ if (mutator(&stmt->limitCount, context))
+ return true;
+ if (mutator(&stmt->lockingClause, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ if (mutator(&stmt->larg, context))
+ return true;
+ if (mutator(&stmt->rarg, context))
+ return true;
+ }
+ break;
+ case T_A_Expr:
+ {
+ A_Expr *expr = (A_Expr *) node;
+
+ if (mutator(&expr->lexpr, context))
+ return true;
+ if (mutator(&expr->rexpr, context))
+ return true;
+ /* operator name is deemed uninteresting */
+ }
+ break;
+ case T_BoolExpr:
+ {
+ BoolExpr *expr = (BoolExpr *) node;
+
+ if (mutator(&expr->args, context))
+ return true;
+ }
+ break;
+ case T_ColumnRef:
+ /* we assume the fields contain nothing interesting */
+ break;
+ case T_FuncCall:
+ {
+ FuncCall *fcall = (FuncCall *) node;
+
+ if (mutator(&fcall->args, context))
+ return true;
+ if (mutator(&fcall->agg_order, context))
+ return true;
+ if (mutator(&fcall->agg_filter, context))
+ return true;
+ if (mutator(&fcall->over, context))
+ return true;
+ /* function name is deemed uninteresting */
+ }
+ break;
+ case T_NamedArgExpr:
+ return mutator(&((NamedArgExpr *) node)->arg, context);
+ case T_A_Indices:
+ {
+ A_Indices *indices = (A_Indices *) node;
+
+ if (mutator(&indices->lidx, context))
+ return true;
+ if (mutator(&indices->uidx, context))
+ return true;
+ }
+ break;
+ case T_A_Indirection:
+ {
+ A_Indirection *indir = (A_Indirection *) node;
+
+ if (mutator(&indir->arg, context))
+ return true;
+ if (mutator(&indir->indirection, context))
+ return true;
+ }
+ break;
+ case T_A_ArrayExpr:
+ return mutator(&((A_ArrayExpr *) node)->elements, context);
+ case T_ResTarget:
+ {
+ ResTarget *rt = (ResTarget *) node;
+
+ if (mutator(&rt->indirection, context))
+ return true;
+ if (mutator(&rt->val, context))
+ return true;
+ }
+ break;
+ case T_MultiAssignRef:
+ return mutator(&((MultiAssignRef *) node)->source, context);
+ case T_TypeCast:
+ {
+ TypeCast *tc = (TypeCast *) node;
+
+ if (mutator(&tc->arg, context))
+ return true;
+ if (mutator(&tc->typeName, context))
+ return true;
+ }
+ break;
+ case T_CollateClause:
+ return mutator(&((CollateClause *) node)->arg, context);
+ case T_SortBy:
+ return mutator(&((SortBy *) node)->node, context);
+ case T_WindowDef:
+ {
+ WindowDef *wd = (WindowDef *) node;
+
+ if (mutator(&wd->partitionClause, context))
+ return true;
+ if (mutator(&wd->orderClause, context))
+ return true;
+ if (mutator(&wd->startOffset, context))
+ return true;
+ if (mutator(&wd->endOffset, context))
+ return true;
+ }
+ break;
+ case T_RangeSubselect:
+ {
+ RangeSubselect *rs = (RangeSubselect *) node;
+
+ if (mutator(&rs->subquery, context))
+ return true;
+ if (mutator(&rs->alias, context))
+ return true;
+ }
+ break;
+ case T_RangeFunction:
+ {
+ RangeFunction *rf = (RangeFunction *) node;
+
+ if (mutator(&rf->functions, context))
+ return true;
+ if (mutator(&rf->alias, context))
+ return true;
+ if (mutator(&rf->coldeflist, context))
+ return true;
+ }
+ break;
+ case T_RangeTableSample:
+ {
+ RangeTableSample *rts = (RangeTableSample *) node;
+
+ if (mutator(&rts->relation, context))
+ return true;
+ /* method name is deemed uninteresting */
+ if (mutator(&rts->args, context))
+ return true;
+ if (mutator(&rts->repeatable, context))
+ return true;
+ }
+ break;
+ case T_TypeName:
+ {
+ TypeName *tn = (TypeName *) node;
+
+ if (mutator(&tn->typmods, context))
+ return true;
+ if (mutator(&tn->arrayBounds, context))
+ return true;
+ /* type name itself is deemed uninteresting */
+ }
+ break;
+ case T_ColumnDef:
+ {
+ ColumnDef *coldef = (ColumnDef *) node;
+
+ if (mutator(&coldef->typeName, context))
+ return true;
+ if (mutator(&coldef->raw_default, context))
+ return true;
+ if (mutator(&coldef->collClause, context))
+ return true;
+ /* for now, constraints are ignored */
+ }
+ break;
+ case T_IndexElem:
+ {
+ IndexElem *indelem = (IndexElem *) node;
+
+ if (mutator(&indelem->expr, context))
+ return true;
+ /* collation and opclass names are deemed uninteresting */
+ }
+ break;
+ case T_GroupingSet:
+ return mutator(&((GroupingSet *) node)->content, context);
+ case T_LockingClause:
+ return mutator(&((LockingClause *) node)->lockedRels, context);
+ case T_XmlSerialize:
+ {
+ XmlSerialize *xs = (XmlSerialize *) node;
+
+ if (mutator(&xs->expr, context))
+ return true;
+ if (mutator(&xs->typeName, context))
+ return true;
+ }
+ break;
+ case T_WithClause:
+ return mutator(&((WithClause *) node)->ctes, context);
+ case T_InferClause:
+ {
+ InferClause *stmt = (InferClause *) node;
+
+ if (mutator(&stmt->indexElems, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ }
+ break;
+ case T_OnConflictClause:
+ {
+ OnConflictClause *stmt = (OnConflictClause *) node;
+
+ if (mutator(&stmt->infer, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ }
+ break;
+ case T_CommonTableExpr:
+ return mutator(&((CommonTableExpr *) node)->ctequery, context);
+ default:
+ return true;
+ }
+ return false;
+}
+
+/*
* planstate_tree_walker --- walk plan state trees
*
* The walker has already visited the current node, and so we need only
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3055b48..0f7aaac 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -42,11 +42,13 @@
#include "catalog/pg_type.h"
#include "commands/async.h"
#include "commands/prepare.h"
+#include "commands/defrem.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "nodes/print.h"
+#include "nodes/nodeFuncs.h"
#include "optimizer/planner.h"
#include "pgstat.h"
#include "pg_trace.h"
@@ -73,6 +75,7 @@
#include "utils/snapmgr.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
+#include "utils/int8.h"
#include "mb/pg_wchar.h"
@@ -188,7 +191,16 @@ static bool IsTransactionStmtList(List *pstmts);
static void drop_unnamed_stmt(void);
static void SigHupHandler(SIGNAL_ARGS);
static void log_disconnections(int code, Datum arg);
-
+static bool exec_cached_query(const char* query, List *parsetree_list);
+static void exec_prepared_plan(Portal portal, const char *portal_name, long max_rows, CommandDest dest);
+static CachedPlanSource *prepare_cached_plan(const char *query_string,/* string to execute */
+ const char *stmt_name, /* name for prepared stmt */
+ Oid *paramTypes, /* parameter types */
+ int numParams, /* number of parameters */
+ List *parsetree_list, /* raw parse tree */
+ MemoryContext unnamed_stmt_context, /* memory context for unnamed statements */
+ bool noerror,
+ bool* requires_snapshot);
/* ----------------------------------------------------------------
* routines to obtain user input
@@ -958,6 +970,14 @@ exec_simple_query(const char *query_string)
isTopLevel = (list_length(parsetree_list) == 1);
/*
+ * Try to find cached plan
+ */
+ if (isTopLevel && autoprepare_threshold != 0 && exec_cached_query(query_string, parsetree_list))
+ {
+ return;
+ }
+
+ /*
* Run through the raw parsetree(s) and process each one.
*/
foreach(parsetree_item, parsetree_list)
@@ -1202,12 +1222,9 @@ exec_parse_message(const char *query_string, /* string to execute */
MemoryContext unnamed_stmt_context = NULL;
MemoryContext oldcontext;
List *parsetree_list;
- RawStmt *raw_parse_tree;
- const char *commandTag;
- List *querytree_list;
- CachedPlanSource *psrc;
bool is_named;
bool save_log_statement_stats = log_statement_stats;
+ bool requires_snapshot;
char msec_str[32];
/*
@@ -1281,144 +1298,14 @@ exec_parse_message(const char *query_string, /* string to execute */
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot insert multiple commands into a prepared statement")));
- if (parsetree_list != NIL)
- {
- Query *query;
- bool snapshot_set = false;
- int i;
-
- raw_parse_tree = castNode(RawStmt, linitial(parsetree_list));
-
- /*
- * Get the command name for possible use in status display.
- */
- commandTag = CreateCommandTag(raw_parse_tree->stmt);
-
- /*
- * If we are in an aborted transaction, reject all commands except
- * COMMIT/ROLLBACK. It is important that this test occur before we
- * try to do parse analysis, rewrite, or planning, since all those
- * phases try to do database accesses, which may fail in abort state.
- * (It might be safe to allow some additional utility commands in this
- * state, but not many...)
- */
- if (IsAbortedTransactionBlockState() &&
- !IsTransactionExitStmt(raw_parse_tree->stmt))
- ereport(ERROR,
- (errcode(ERRCODE_IN_FAILED_SQL_TRANSACTION),
- errmsg("current transaction is aborted, "
- "commands ignored until end of transaction block"),
- errdetail_abort()));
-
- /*
- * Create the CachedPlanSource before we do parse analysis, since it
- * needs to see the unmodified raw parse tree.
- */
- psrc = CreateCachedPlan(raw_parse_tree, query_string, commandTag);
-
- /*
- * Set up a snapshot if parse analysis will need one.
- */
- if (analyze_requires_snapshot(raw_parse_tree))
- {
- PushActiveSnapshot(GetTransactionSnapshot());
- snapshot_set = true;
- }
-
- /*
- * Analyze and rewrite the query. Note that the originally specified
- * parameter set is not required to be complete, so we have to use
- * parse_analyze_varparams().
- */
- if (log_parser_stats)
- ResetUsage();
-
- query = parse_analyze_varparams(raw_parse_tree,
- query_string,
- ¶mTypes,
- &numParams);
-
- /*
- * Check all parameter types got determined.
- */
- for (i = 0; i < numParams; i++)
- {
- Oid ptype = paramTypes[i];
-
- if (ptype == InvalidOid || ptype == UNKNOWNOID)
- ereport(ERROR,
- (errcode(ERRCODE_INDETERMINATE_DATATYPE),
- errmsg("could not determine data type of parameter $%d",
- i + 1)));
- }
-
- if (log_parser_stats)
- ShowUsage("PARSE ANALYSIS STATISTICS");
-
- querytree_list = pg_rewrite_query(query);
-
- /* Done with the snapshot used for parsing */
- if (snapshot_set)
- PopActiveSnapshot();
- }
- else
- {
- /* Empty input string. This is legal. */
- raw_parse_tree = NULL;
- commandTag = NULL;
- psrc = CreateCachedPlan(raw_parse_tree, query_string, commandTag);
- querytree_list = NIL;
- }
-
- /*
- * CachedPlanSource must be a direct child of MessageContext before we
- * reparent unnamed_stmt_context under it, else we have a disconnected
- * circular subgraph. Klugy, but less so than flipping contexts even more
- * above.
+ /*
+ * Create cached plan, analyze parameters and save cached plan
*/
- if (unnamed_stmt_context)
- MemoryContextSetParent(psrc->context, MessageContext);
-
- /* Finish filling in the CachedPlanSource */
- CompleteCachedPlan(psrc,
- querytree_list,
- unnamed_stmt_context,
- paramTypes,
- numParams,
- NULL,
- NULL,
- CURSOR_OPT_PARALLEL_OK, /* allow parallel mode */
- true); /* fixed result */
-
- /* If we got a cancel signal during analysis, quit */
- CHECK_FOR_INTERRUPTS();
-
- if (is_named)
- {
- /*
- * Store the query as a prepared statement.
- */
- StorePreparedStatement(stmt_name, psrc, false);
- }
- else
- {
- /*
- * We just save the CachedPlanSource into unnamed_stmt_psrc.
- */
- SaveCachedPlan(psrc);
- unnamed_stmt_psrc = psrc;
- }
+ prepare_cached_plan(query_string, stmt_name, paramTypes, numParams, parsetree_list, unnamed_stmt_context, false, &requires_snapshot);
MemoryContextSwitchTo(oldcontext);
/*
- * We do NOT close the open transaction command here; that only happens
- * when the client sends Sync. Instead, do CommandCounterIncrement just
- * in case something happened during parse/plan.
- */
- CommandCounterIncrement();
-
- /*
* Send ParseComplete.
*/
if (whereToSendOutput == DestRemote)
@@ -1841,9 +1728,28 @@ exec_bind_message(StringInfo input_message)
static void
exec_execute_message(const char *portal_name, long max_rows)
{
- CommandDest dest;
+ Portal portal = GetPortalByName(portal_name);
+ CommandDest dest = whereToSendOutput;
+
+ /* Adjust destination to tell printtup.c what to do */
+ if (dest == DestRemote)
+ dest = DestRemoteExecute;
+
+ if (!PortalIsValid(portal))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_CURSOR),
+ errmsg("portal \"%s\" does not exist", portal_name)));
+
+ exec_prepared_plan(portal, portal_name, max_rows, dest);
+}
+
+/*
+ * Execute prepared plan.
+ */
+static void
+exec_prepared_plan(Portal portal, const char *portal_name, long max_rows, CommandDest dest)
+{
DestReceiver *receiver;
- Portal portal;
bool completed;
char completionTag[COMPLETION_TAG_BUFSIZE];
const char *sourceText;
@@ -1855,17 +1761,6 @@ exec_execute_message(const char *portal_name, long max_rows)
bool was_logged = false;
char msec_str[32];
- /* Adjust destination to tell printtup.c what to do */
- dest = whereToSendOutput;
- if (dest == DestRemote)
- dest = DestRemoteExecute;
-
- portal = GetPortalByName(portal_name);
- if (!PortalIsValid(portal))
- ereport(ERROR,
- (errcode(ERRCODE_UNDEFINED_CURSOR),
- errmsg("portal \"%s\" does not exist", portal_name)));
-
/*
* If the original query was a null string, just return
* EmptyQueryResponse.
@@ -1928,7 +1823,7 @@ exec_execute_message(const char *portal_name, long max_rows)
* context, because that may get deleted if portal contains VACUUM).
*/
receiver = CreateDestReceiver(dest);
- if (dest == DestRemoteExecute)
+ if (dest == DestRemoteExecute || dest == DestRemote)
SetRemoteDestReceiverParams(receiver, portal);
/*
@@ -4507,3 +4402,815 @@ log_disconnections(int code, Datum arg)
port->user_name, port->database_name, port->remote_host,
port->remote_port[0] ? " port=" : "", port->remote_port)));
}
+
+/*
+ * Create cached plan, analyze parameters and save cached plan
+ */
+static CachedPlanSource *prepare_cached_plan(const char *query_string,/* string to execute */
+ const char *stmt_name, /* name for prepared stmt */
+ Oid *paramTypes, /* parameter types */
+ int numParams, /* number of parameters */
+ List *parsetree_list, /* raw parse tree */
+ MemoryContext unnamed_stmt_context, /* memory context for unnamed statements */
+ bool noerror, /* do not report error in case of unresolved parameters */
+ bool* requires_snapshot) /* parse analysis requires snapshot */
+{
+ CachedPlanSource *psrc;
+ RawStmt *raw_parse_tree;
+ const char *commandTag;
+ List *querytree_list;
+
+ *requires_snapshot = false;
+
+ if (parsetree_list != NIL)
+ {
+ Query *query;
+ bool snapshot_set = false;
+ int i;
+
+ raw_parse_tree = castNode(RawStmt, linitial(parsetree_list));
+
+ /*
+ * Get the command name for possible use in status display.
+ */
+ commandTag = CreateCommandTag(raw_parse_tree->stmt);
+
+ /*
+ * If we are in an aborted transaction, reject all commands except
+ * COMMIT/ROLLBACK. It is important that this test occur before we
+ * try to do parse analysis, rewrite, or planning, since all those
+ * phases try to do database accesses, which may fail in abort state.
+ * (It might be safe to allow some additional utility commands in this
+ * state, but not many...)
+ */
+ if (IsAbortedTransactionBlockState() &&
+ !IsTransactionExitStmt(raw_parse_tree->stmt))
+ ereport(ERROR,
+ (errcode(ERRCODE_IN_FAILED_SQL_TRANSACTION),
+ errmsg("current transaction is aborted, "
+ "commands ignored until end of transaction block"),
+ errdetail_abort()));
+
+ /*
+ * Create the CachedPlanSource before we do parse analysis, since it
+ * needs to see the unmodified raw parse tree.
+ */
+ psrc = CreateCachedPlan(raw_parse_tree, query_string, commandTag);
+
+ /*
+ * Set up a snapshot if parse analysis will need one.
+ */
+ if (analyze_requires_snapshot(raw_parse_tree))
+ {
+ PushActiveSnapshot(GetTransactionSnapshot());
+ *requires_snapshot = snapshot_set = true;
+ }
+
+ /*
+ * Analyze and rewrite the query. Note that the originally specified
+ * parameter set is not required to be complete, so we have to use
+ * parse_analyze_varparams().
+ */
+ if (log_parser_stats)
+ ResetUsage();
+
+ query = parse_analyze_varparams(raw_parse_tree,
+ query_string,
+ ¶mTypes,
+ &numParams);
+
+ /*
+ * Check all parameter types got determined.
+ */
+ for (i = 0; i < numParams; i++)
+ {
+ Oid ptype = paramTypes[i];
+
+ if (ptype == InvalidOid || ptype == UNKNOWNOID)
+ {
+ if (noerror)
+ {
+ if (snapshot_set)
+ PopActiveSnapshot();
+ return NULL;
+ }
+ else
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_INDETERMINATE_DATATYPE),
+ errmsg("could not determine data type of parameter $%d",
+ i + 1)));
+ }
+ }
+ }
+
+ if (log_parser_stats)
+ ShowUsage("PARSE ANALYSIS STATISTICS");
+
+ querytree_list = pg_rewrite_query(query);
+
+ /* Done with the snapshot used for parsing */
+ if (snapshot_set)
+ PopActiveSnapshot();
+ }
+ else
+ {
+ /* Empty input string. This is legal. */
+ raw_parse_tree = NULL;
+ commandTag = NULL;
+ psrc = CreateCachedPlan(raw_parse_tree, query_string, commandTag);
+ querytree_list = NIL;
+ }
+
+ /*
+ * CachedPlanSource must be a direct child of MessageContext before we
+ * reparent unnamed_stmt_context under it, else we have a disconnected
+ * circular subgraph. Klugy, but less so than flipping contexts even more
+ * above.
+ */
+ if (unnamed_stmt_context)
+ MemoryContextSetParent(psrc->context, MessageContext);
+
+ /* Finish filling in the CachedPlanSource */
+ CompleteCachedPlan(psrc,
+ querytree_list,
+ unnamed_stmt_context,
+ paramTypes,
+ numParams,
+ NULL,
+ NULL,
+ CURSOR_OPT_PARALLEL_OK, /* allow parallel mode */
+ true); /* fixed result */
+
+ /* If we got a cancel signal during analysis, quit */
+ CHECK_FOR_INTERRUPTS();
+
+ if (stmt_name[0] != '\0')
+ {
+ /*
+ * Store the query as a prepared statement.
+ */
+ StorePreparedStatement(stmt_name, psrc, false);
+ }
+ else
+ {
+ /*
+ * We just save the CachedPlanSource into unnamed_stmt_psrc.
+ */
+ SaveCachedPlan(psrc);
+ unnamed_stmt_psrc = psrc;
+ }
+
+ /*
+ * We do NOT close the open transaction command here; that only happens
+ * when the client sends Sync. Instead, do CommandCounterIncrement just
+ * in case something happened during parse/plan.
+ */
+ CommandCounterIncrement();
+
+ return psrc;
+}
+
+/*
+ * Autoprepare implementation.
+ * Autoprepare consists of raw parse tree mutator, hash table of cached plans and exec_cached_query function
+ * which combines exec_parse_message + exec_bind_message + exec_execute_message
+ */
+
+/*
+ * Mapping between parameters and replaced literals
+ */
+typedef struct ParamBinding
+{
+ A_Const* literal; /* Original literal */
+ ParamRef* param; /* Constructed parameter reference */
+ Node** ref; /* Pointer to pointer to literal node (used to revert raw parse tree update) */
+ Oid type; /* Parameter type */
+ struct ParamBinding* next; /* L1-list of query parameter bindings */
+} ParamBinding;
+
+/*
+ * Plan cache entry
+ */
+typedef struct
+{
+ Node* parse_tree; /* tree is used as hash key */
+ dlist_node lru; /* double linked list to implement LRU */
+ int64 exec_count; /* counter of execution of this query */
+ CachedPlanSource* plan;
+ uint32 hash; /* hash calculated for this parsed tree */
+ Oid* param_types;/* types of parameters */
+ int n_params; /* number of parameters extracted for this query */
+ int16 format; /* portal output format */
+ bool disable_autoprepare; /* disable preparing of this query */
+} plan_cache_entry;
+
+static uint32 plan_cache_hash_fn(const void *key, Size keysize)
+{
+ return ((plan_cache_entry*)key)->hash;
+}
+
+static int plan_cache_match_fn(const void *key1, const void *key2, Size keysize)
+{
+ plan_cache_entry* e1 = (plan_cache_entry*)key1;
+ plan_cache_entry* e2 = (plan_cache_entry*)key2;
+
+ return equal(e1->parse_tree, e2->parse_tree)
+ && memcmp(e1->param_types, e2->param_types, sizeof(Oid)*e1->n_params) == 0 ? 0 : 1;
+}
+
+static void* plan_cache_keycopy_fn(void *dest, const void *src, Size keysize)
+{
+ plan_cache_entry* dst_entry = (plan_cache_entry*)dest;
+ plan_cache_entry* src_entry = (plan_cache_entry*)src;
+ dst_entry->parse_tree = copyObject(src_entry->parse_tree);
+ dst_entry->param_types = palloc(src_entry->n_params*sizeof(Oid));
+ dst_entry->n_params = src_entry->n_params;
+ memcpy(dst_entry->param_types, src_entry->param_types, src_entry->n_params*sizeof(Oid));
+ dst_entry->hash = src_entry->hash;
+ return dest;
+}
+
+#define PLAN_CACHE_SIZE 113
+
+/*
+ * Plan cache access statistic
+ */
+size_t autoprepare_hits;
+size_t autoprepare_misses;
+size_t autoprepare_cached_plans;
+
+/*
+ * Context for raw_expression_tree_mutator
+ */
+typedef struct {
+ int n_params; /* Number of extracted parameters */
+ uint32 hash; /* We calculate hash for parse tree during plan traversal */
+ ParamBinding** param_list_tail; /* pointer to last element "next" field address, used to construct L1 list of parameters */
+} GeneralizerCtx;
+
+/*
+ * Check if expression is constant (used to eliminate substitution of literals with parameters in such expressions
+ */
+static bool is_constant_expression(Node* node)
+{
+ return node != NULL
+ && (IsA(node, A_Const)
+ || (IsA(node, A_Expr)
+ && is_constant_expression(((A_Expr*)node)->lexpr)
+ && is_constant_expression(((A_Expr*)node)->rexpr)));
+}
+
+/*
+ * Infer type of literal expression. Null literals should not be replaced with parameters.
+ */
+static Oid get_literal_type(Value* val)
+{
+ int64 val64;
+ switch (val->type)
+ {
+ case T_Integer:
+ return INT4OID;
+ case T_Float:
+ /* could be an oversize integer as well as a float ... */
+ if (scanint8(strVal(val), true, &val64))
+ {
+ /*
+ * It might actually fit in int32. Probably only INT_MIN can
+ * occur, but we'll code the test generally just to be sure.
+ */
+ int32 val32 = (int32) val64;
+ return (val64 == (int64)val32) ? INT4OID : INT8OID;
+ }
+ else
+ {
+ return NUMERICOID;
+ }
+ case T_BitString:
+ return BITOID;
+ case T_String:
+ return UNKNOWNOID;
+ default:
+ Assert(false);
+ return InvalidOid;
+ }
+}
+
+static Datum get_param_value(Oid type, Value* val)
+{
+ if (val->type == T_Integer)
+ {
+ /*
+ * Integer constant
+ */
+ return Int32GetDatum((int32)val->val.ival);
+ }
+ else
+ {
+ /*
+ * Convert from string literal
+ */
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(type, &typinput, &typioparam);
+ return OidInputFunctionCall(typinput, val->val.str, typioparam, -1);
+ }
+}
+
+
+/*
+ * Callback for raw_expression_tree_mutator performing substitution of literals with parameters
+ */
+static bool
+query_plan_generalizer(Node** ref, void *context)
+{
+ Node* node = *ref;
+ GeneralizerCtx* ctx = (GeneralizerCtx*)context;
+ if (node == NULL)
+ {
+ return false;
+ }
+ /*
+ * Calculate hash for parse tree. We consider only node tags here, precise comparison of trees is done using equal() function.
+ * Here we calculate hash for original (unpatched) tree, without ParamRef nodes.
+ * It is non principle, because hash calculation doesn't take in account types and values of Const nodes. So the same generalized queries
+ * will have the same hash value. There are about 1000 different nodes tags, this is why we rotate hash on 10 bits.
+ */
+ ctx->hash = (ctx->hash << 10) ^ (ctx->hash >> 22) ^ nodeTag(node);
+
+ switch (nodeTag(node))
+ {
+ case T_A_Expr:
+ {
+ /*
+ * Do not perform substitution of literals in constant expression (which is likely to be the same for all queries and optimized by compiler)
+ */
+ if (!is_constant_expression(node))
+ {
+ A_Expr *expr = (A_Expr *) node;
+ if (query_plan_generalizer((Node**)&expr->lexpr, context))
+ return true;
+ if (query_plan_generalizer((Node**)&expr->rexpr, context))
+ return true;
+ }
+ break;
+ }
+ case T_A_Const:
+ {
+ /*
+ * Do substitution of literals with parameters here
+ */
+ A_Const* literal = (A_Const*)node;
+ if (literal->val.type != T_Null)
+ {
+ /*
+ * Do not substitute null literals with parameters
+ */
+ ParamBinding* cp = palloc(sizeof(ParamBinding));
+ ParamRef* param = makeNode(ParamRef);
+ param->number = ++ctx->n_params;
+ param->location = literal->location;
+ cp->ref = ref;
+ cp->param = param;
+ cp->literal = literal;
+ cp->type = get_literal_type(&literal->val);
+ *ctx->param_list_tail = cp;
+ ctx->param_list_tail = &cp->next;
+ *ref = (Node*)param;
+ }
+ break;
+ }
+ case T_SelectStmt:
+ {
+ /*
+ * Substitute literals only in target list, WHERE, VALUES and WITH clause,
+ * skipping target and from lists, which is unlikely contains some parameterized values
+ */
+ SelectStmt *stmt = (SelectStmt *) node;
+ if (query_plan_generalizer((Node**)&stmt->targetList, context))
+ return true;
+ if (query_plan_generalizer((Node**)&stmt->whereClause, context))
+ return true;
+ if (query_plan_generalizer((Node**)&stmt->valuesLists, context))
+ return true;
+ if (query_plan_generalizer((Node**)&stmt->withClause, context))
+ return true;
+ if (query_plan_generalizer((Node**)&stmt->larg, context))
+ return true;
+ if (query_plan_generalizer((Node**)&stmt->rarg, context))
+ return true;
+ break;
+ }
+ case T_TypeName:
+ case T_SortGroupClause:
+ case T_SortBy:
+ case T_A_ArrayExpr:
+ case T_TypeCast:
+ /*
+ * Literals in this clauses should not be replaced with parameters
+ */
+ break;
+ default:
+ /*
+ * Default traversal. raw_expression_tree_mutator returns true for all not recognized nodes, for example right now
+ * all transaction control statements are not covered by raw_expression_tree_mutator and so will not auto prepared.
+ * My experiments show that effect of non-preparing start/commit transaction statements is positive.
+ */
+ return raw_expression_tree_mutator(node, query_plan_generalizer, context);
+ }
+ return false;
+}
+
+/*
+ * Restore original parse tree, replacing all ParamRef back with Const nodes.
+ * Such undo operation seems to be more efficient than copying the whole parse tree by raw_expression_tree_mutator
+ */
+static void undo_query_plan_changes(ParamBinding* cp)
+{
+ while (cp != NULL) {
+ *cp->ref = (Node*)cp->literal;
+ cp = cp->next;
+ }
+ autoprepare_misses += 1;
+}
+
+/*
+ * Callback for raw_expression_tree_walker dropping parse tree
+ */
+static bool drop_tree_node(Node* node, void* context)
+{
+ if (node) {
+ raw_expression_tree_walker(node, drop_tree_node, NULL);
+ pfree(node);
+ }
+ return false;
+}
+
+/*
+ * Location of converted literal in query.
+ * Used for precise error reporting (line number)
+ */
+static int param_location;
+
+/*
+ * Error callback adding information about error location
+ */
+static void
+prepare_error_callback(void *arg)
+{
+ CachedPlanSource *psrc = (CachedPlanSource*)arg;
+ /* And pass it to the ereport mechanism */
+ if (geterrcode() != ERRCODE_QUERY_CANCELED) {
+ int pos = pg_mbstrlen_with_len(psrc->query_string, param_location) + 1;
+ (void)errposition(pos);
+ }
+}
+
+/*
+ * Try to generalize query, find cached plan for it and execute
+ */
+static bool exec_cached_query(const char *query_string, List *parsetree_list)
+{
+ int n_params;
+ plan_cache_entry *entry;
+ bool found;
+ MemoryContext old_context;
+ CachedPlanSource *psrc;
+ ParamListInfo params;
+ int paramno;
+ CachedPlan *cplan;
+ Portal portal;
+ bool snapshot_set = false;
+ GeneralizerCtx ctx;
+ ParamBinding* binding;
+ ParamBinding* binding_list;
+ plan_cache_entry pattern;
+ Oid* param_types;
+ RawStmt *raw_parse_tree;
+
+ static HTAB* plan_cache; /* hash table for plan cache */
+ static dlist_head lru; /* LRU L2-list for cached queries */
+ static MemoryContext plan_cache_context; /* memory context used for plan cache */
+
+ raw_parse_tree = castNode(RawStmt, linitial(parsetree_list));
+
+ /*
+ * Substitute literals with parameters and calculate hash for raw parse tree
+ */
+ ctx.param_list_tail = &binding_list;
+ ctx.n_params = 0;
+ ctx.hash = 0;
+ if (query_plan_generalizer(&raw_parse_tree->stmt, &ctx))
+ {
+ *ctx.param_list_tail = NULL;
+ undo_query_plan_changes(binding_list);
+ return false;
+ }
+ *ctx.param_list_tail = NULL;
+ n_params = ctx.n_params;
+
+ /*
+ * Extract array of parameters types: it is needed for cached plan lookup
+ */
+ param_types = (Oid*)palloc(sizeof(Oid)*n_params);
+ for (paramno = 0, binding = binding_list; paramno < n_params; paramno++, binding = binding->next)
+ {
+ param_types[paramno] = binding->type;
+ }
+
+ /*
+ * Construct plan cache context if not constructed yet.
+ */
+ if (plan_cache_context == NULL)
+ {
+ plan_cache_context = AllocSetContextCreate(TopMemoryContext,
+ "plan cache context",
+ ALLOCSET_DEFAULT_SIZES);
+ }
+ /* Manipulations with hash table are performed in plan_cache_context memory context */
+ old_context = MemoryContextSwitchTo(plan_cache_context);
+
+ /*
+ * Initialize hash table if not initialized yet
+ */
+ if (plan_cache == NULL)
+ {
+ static HASHCTL info;
+ info.keysize = sizeof(plan_cache_entry);
+ info.entrysize = sizeof(plan_cache_entry);
+ info.hash = plan_cache_hash_fn;
+ info.match = plan_cache_match_fn;
+ info.keycopy = plan_cache_keycopy_fn;
+ plan_cache = hash_create("plan_cache", autoprepare_limit != 0 ? autoprepare_limit : PLAN_CACHE_SIZE,
+ &info, HASH_ELEM | HASH_FUNCTION | HASH_COMPARE | HASH_KEYCOPY);
+ dlist_init(&lru);
+ }
+
+ /*
+ * Lookup generalized query
+ */
+ pattern.parse_tree = raw_parse_tree->stmt;
+ pattern.hash = ctx.hash;
+ pattern.n_params = n_params;
+ pattern.param_types = param_types;
+ entry = (plan_cache_entry*)hash_search(plan_cache, &pattern, HASH_ENTER, &found);
+ if (!found)
+ {
+ /* Check number of cached queries */
+ if (++autoprepare_cached_plans > autoprepare_limit && autoprepare_limit != 0)
+ {
+ /* Drop least recently accessed query */
+ plan_cache_entry* victim = dlist_container(plan_cache_entry, lru, lru.head.prev);
+ Node* dropped_tree = victim->parse_tree;
+ dlist_delete(&victim->lru);
+ if (victim->plan)
+ {
+ DropCachedPlan(victim->plan);
+ }
+ pfree(victim->param_types);
+ hash_search(plan_cache, victim, HASH_REMOVE, NULL);
+ raw_expression_tree_walker(dropped_tree, drop_tree_node, NULL);
+ autoprepare_cached_plans -= 1;
+ }
+ entry->exec_count = 0;
+ entry->plan = NULL;
+ entry->disable_autoprepare = false;
+ }
+ else
+ {
+ dlist_delete(&entry->lru); /* accessed entry will be moved to the head of LRU list */
+ if (entry->plan != NULL && !entry->plan->is_valid)
+ {
+ /* Drop invalidated plan: it will be reconstructed later */
+ DropCachedPlan(entry->plan);
+ entry->plan = NULL;
+ }
+ }
+ dlist_insert_after(&lru.head, &entry->lru); /* prepend entry to the head of LRU list */
+ MemoryContextSwitchTo(old_context); /* Done with plan_cache_context memory context */
+
+ /*
+ * Prepare query only when it is executed more than autoprepare_threshold times
+ */
+ if (entry->disable_autoprepare || entry->exec_count++ < autoprepare_threshold)
+ {
+ undo_query_plan_changes(binding_list);
+ return false;
+ }
+ if (entry->plan == NULL)
+ {
+ bool requires_snapshot;
+
+ /*
+ * Switch to appropriate context for preparing plan.
+ */
+ old_context = MemoryContextSwitchTo(MessageContext);
+
+ /*
+ * Create cached plan, analyze parameters and save cached plan
+ */
+ PG_TRY();
+ {
+ psrc = prepare_cached_plan(query_string, "", param_types, n_params, parsetree_list, NULL, true, &requires_snapshot);
+ }
+ PG_CATCH();
+ {
+ /*
+ * In case of analyze errors revert back to original query processing
+ * and disable autoprepare for this query to avoid such problems in future.
+ */
+ FlushErrorState();
+ if (requires_snapshot)
+ {
+ PopActiveSnapshot();
+ }
+ psrc = NULL;
+ }
+ PG_END_TRY();
+ MemoryContextSwitchTo(old_context);
+
+ if (psrc == NULL)
+ {
+ /* Failed to resolve parameter type */
+ undo_query_plan_changes(binding_list);
+ entry->disable_autoprepare = true;
+ return false;
+ }
+
+ unnamed_stmt_psrc = NULL; /* prevent destruction of prepared plan by drop_unnamed_stmt */
+ entry->plan = psrc;
+
+ /*
+ * Determine output format
+ */
+ entry->format = 0; /* TEXT is default */
+ if (IsA(raw_parse_tree->stmt, FetchStmt))
+ {
+ FetchStmt *stmt = (FetchStmt *)raw_parse_tree->stmt;
+
+ if (!stmt->ismove)
+ {
+ Portal fportal = GetPortalByName(stmt->portalname);
+
+ if (PortalIsValid(fportal) &&
+ (fportal->cursorOptions & CURSOR_OPT_BINARY))
+ entry->format = 1; /* BINARY */
+ }
+ }
+ }
+ else
+ {
+ /* Plan found */
+ psrc = entry->plan;
+ Assert(n_params == entry->n_params);
+ }
+
+ /*
+ * If we are in aborted transaction state, the only portals we can
+ * actually run are those containing COMMIT or ROLLBACK commands. We
+ * disallow binding anything else to avoid problems with infrastructure
+ * that expects to run inside a valid transaction. We also disallow
+ * binding any parameters, since we can't risk calling user-defined I/O
+ * functions.
+ */
+ if (IsAbortedTransactionBlockState() &&
+ (!IsTransactionExitStmt(psrc->raw_parse_tree->stmt) ||
+ n_params != 0))
+ ereport(ERROR,
+ (errcode(ERRCODE_IN_FAILED_SQL_TRANSACTION),
+ errmsg("current transaction is aborted, "
+ "commands ignored until end of transaction block"),
+ errdetail_abort()));
+
+ /*
+ * Create unnamed portal to run the query or queries in. If there
+ * already is one, silently drop it.
+ */
+ portal = CreatePortal("", true, true);
+ /* Don't display the portal in pg_cursors */
+ portal->visible = false;
+
+ /*
+ * Prepare to copy stuff into the portal's memory context. We do all this
+ * copying first, because it could possibly fail (out-of-memory) and we
+ * don't want a failure to occur between GetCachedPlan and
+ * PortalDefineQuery; that would result in leaking our plancache refcount.
+ */
+ old_context = MemoryContextSwitchTo(PortalGetHeapMemory(portal));
+
+ /* Copy the plan's query string into the portal */
+ query_string = pstrdup(psrc->query_string);
+
+ /*
+ * Set a snapshot if we have parameters to fetch (since the input
+ * functions might need it) or the query isn't a utility command (and
+ * hence could require redoing parse analysis and planning). We keep the
+ * snapshot active till we're done, so that plancache.c doesn't have to
+ * take new ones.
+ */
+ if (n_params > 0 ||
+ (psrc->raw_parse_tree &&
+ analyze_requires_snapshot(psrc->raw_parse_tree)))
+ {
+ PushActiveSnapshot(GetTransactionSnapshot());
+ snapshot_set = true;
+ }
+
+ /*
+ * Fetch parameters, if any, and store in the portal's memory context.
+ */
+ if (n_params > 0)
+ {
+ ErrorContextCallback errcallback;
+
+ params = (ParamListInfo) palloc0(offsetof(ParamListInfoData, params) +
+ n_params * sizeof(ParamExternData));
+ params->numParams = n_params;
+
+ /*
+ * Register error callback to precisely report error in case of conversion error while storig parameter value.
+ */
+ errcallback.callback = prepare_error_callback;
+ errcallback.arg = (void *) psrc;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ for (paramno = 0, binding = binding_list;
+ paramno < n_params;
+ paramno++, binding = binding->next)
+ {
+ Oid ptype = psrc->param_types[paramno];
+
+ param_location = binding->literal->location;
+
+ params->params[paramno].isnull = false;
+ params->params[paramno].value = get_param_value(ptype, &binding->literal->val);
+ /*
+ * We mark the params as CONST. This ensures that any custom plan
+ * makes full use of the parameter values.
+ */
+ params->params[paramno].pflags = PARAM_FLAG_CONST;
+ params->params[paramno].ptype = ptype;
+ }
+ error_context_stack = errcallback.previous;
+ }
+ else
+ {
+ params = NULL;
+ }
+
+ /* Done storing stuff in portal's context */
+ MemoryContextSwitchTo(old_context);
+
+ /*
+ * Obtain a plan from the CachedPlanSource. Any cruft from (re)planning
+ * will be generated in MessageContext. The plan refcount will be
+ * assigned to the Portal, so it will be released at portal destruction.
+ */
+ cplan = GetCachedPlan(psrc, params, false);
+
+ /*
+ * Now we can define the portal.
+ *
+ * DO NOT put any code that could possibly throw an error between the
+ * above GetCachedPlan call and here.
+ */
+ PortalDefineQuery(portal,
+ NULL,
+ query_string,
+ psrc->commandTag,
+ cplan->stmt_list,
+ cplan);
+
+ /* Done with the snapshot used for parameter I/O and parsing/planning */
+ if (snapshot_set)
+ {
+ PopActiveSnapshot();
+ }
+
+ /*
+ * And we're ready to start portal execution.
+ */
+ PortalStart(portal, params, 0, InvalidSnapshot);
+
+ /*
+ * Apply the result format requests to the portal.
+ */
+ PortalSetResultFormat(portal, 1, &entry->format);
+
+ /*
+ * Finally execute prepared statement
+ */
+ exec_prepared_plan(portal, "", FETCH_ALL, whereToSendOutput);
+
+ /*
+ * Close down transaction statement, if one is open.
+ */
+ finish_xact_command();
+
+ autoprepare_hits += 1;
+
+ return true;
+}
+
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 8b5f064..5ab83f0 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -476,6 +476,10 @@ int tcp_keepalives_idle;
int tcp_keepalives_interval;
int tcp_keepalives_count;
+
+int autoprepare_threshold;
+int autoprepare_limit;
+
/*
* SSL renegotiation was been removed in PostgreSQL 9.5, but we tolerate it
* being set to zero (meaning never renegotiate) for backward compatibility.
@@ -1960,6 +1964,28 @@ static struct config_int ConfigureNamesInt[] =
check_max_stack_depth, assign_max_stack_depth, NULL
},
+ /*
+ * Threshold for implicit preparing of frequently executed queries
+ */
+ {
+ {"autoprepare_threshold", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Threshold for autopreparing query."),
+ gettext_noop("0 value disables autoprepare.")
+ },
+ &autoprepare_threshold,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+ {
+ {"autoprepare_limit", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Maximal number of autoprepared queries."),
+ gettext_noop("0 means unlimited number of autoprepared queries. Too large number of prepared queries can cause backend memory overflow and slowdown execution speed (because of increased lookup time)")
+ },
+ &autoprepare_limit,
+ 113, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"temp_file_limit", PGC_SUSET, RESOURCES_DISK,
gettext_noop("Limits the total size of all temporary files used by each process."),
diff --git a/src/include/nodes/nodeFuncs.h b/src/include/nodes/nodeFuncs.h
index b6c9b48..1388822 100644
--- a/src/include/nodes/nodeFuncs.h
+++ b/src/include/nodes/nodeFuncs.h
@@ -73,8 +73,11 @@ extern Node *query_or_expression_tree_mutator(Node *node, Node *(*mutator) (),
extern bool raw_expression_tree_walker(Node *node, bool (*walker) (),
void *context);
+extern bool raw_expression_tree_mutator(Node *node, bool (*mutator) (),
+ void *context);
struct PlanState;
extern bool planstate_tree_walker(struct PlanState *planstate, bool (*walker) (),
void *context);
+
#endif /* NODEFUNCS_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 7dd3780..9cd0917 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -251,6 +251,9 @@ extern int client_min_messages;
extern int log_min_duration_statement;
extern int log_temp_files;
+extern int autoprepare_threshold;
+extern int autoprepare_limit;
+
extern int temp_file_limit;
extern int num_temp_buffers;
[text/x-patch] autoprepare-2.patch (40.6K, ../../[email protected]/3-autoprepare-2.patch)
download | inline diff:
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index 6e52eb7..5538c7c 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -3696,6 +3696,454 @@ raw_expression_tree_walker(Node *node,
}
/*
+ * raw_expression_tree_mutator --- transform raw parse tree.
+ *
+ * This function is implementing slightly different approach for tree update than expression_tree_mutator().
+ * Callback is given pointer to pointer to the current node and can update this field instead of returning reference to new node.
+ * It makes it possible to remember changes and easily revert them without extra traversal of the tree.
+ *
+ * This function do not need QTW_DONT_COPY_QUERY flag: it never implicitly copy tree nodes, doing in-place update.
+ *
+ * Like raw_expression_tree_walker, there is no special rule about query
+ * boundaries: we descend to everything that's possibly interesting.
+ *
+ * Currently, the node type coverage here extends only to DML statements
+ * (SELECT/INSERT/UPDATE/DELETE) and nodes that can appear in them, because
+ * this is used mainly during analysis of CTEs, and only DML statements can
+ * appear in CTEs. If some other node is visited, iteration is immediately stopped and true is returned.
+ */
+bool
+raw_expression_tree_mutator(Node *node,
+ bool (*mutator) (),
+ void *context)
+{
+ ListCell *temp;
+
+ /*
+ * The walker has already visited the current node, and so we need only
+ * recurse into any sub-nodes it has.
+ */
+ if (node == NULL)
+ return false;
+
+ /* Guard against stack overflow due to overly complex expressions */
+ check_stack_depth();
+
+ switch (nodeTag(node))
+ {
+ case T_SetToDefault:
+ case T_CurrentOfExpr:
+ case T_Integer:
+ case T_Float:
+ case T_String:
+ case T_BitString:
+ case T_Null:
+ case T_ParamRef:
+ case T_A_Const:
+ case T_A_Star:
+ /* primitive node types with no subnodes */
+ break;
+ case T_Alias:
+ /* we assume the colnames list isn't interesting */
+ break;
+ case T_RangeVar:
+ return mutator(&((RangeVar *) node)->alias, context);
+ case T_GroupingFunc:
+ return mutator(&((GroupingFunc *) node)->args, context);
+ case T_SubLink:
+ {
+ SubLink *sublink = (SubLink *) node;
+
+ if (mutator(&sublink->testexpr, context))
+ return true;
+ /* we assume the operName is not interesting */
+ if (mutator(&sublink->subselect, context))
+ return true;
+ }
+ break;
+ case T_CaseExpr:
+ {
+ CaseExpr *caseexpr = (CaseExpr *) node;
+
+ if (mutator(&caseexpr->arg, context))
+ return true;
+ /* we assume mutator(& doesn't care about CaseWhens, either */
+ foreach(temp, caseexpr->args)
+ {
+ CaseWhen *when = (CaseWhen *) lfirst(temp);
+
+ Assert(IsA(when, CaseWhen));
+ if (mutator(&when->expr, context))
+ return true;
+ if (mutator(&when->result, context))
+ return true;
+ }
+ if (mutator(&caseexpr->defresult, context))
+ return true;
+ }
+ break;
+ case T_RowExpr:
+ /* Assume colnames isn't interesting */
+ return mutator(&((RowExpr *) node)->args, context);
+ case T_CoalesceExpr:
+ return mutator(&((CoalesceExpr *) node)->args, context);
+ case T_MinMaxExpr:
+ return mutator(&((MinMaxExpr *) node)->args, context);
+ case T_XmlExpr:
+ {
+ XmlExpr *xexpr = (XmlExpr *) node;
+
+ if (mutator(&xexpr->named_args, context))
+ return true;
+ /* we assume mutator(& doesn't care about arg_names */
+ if (mutator(&xexpr->args, context))
+ return true;
+ }
+ break;
+ case T_NullTest:
+ return mutator(&((NullTest *) node)->arg, context);
+ case T_BooleanTest:
+ return mutator(&((BooleanTest *) node)->arg, context);
+ case T_JoinExpr:
+ {
+ JoinExpr *join = (JoinExpr *) node;
+
+ if (mutator(&join->larg, context))
+ return true;
+ if (mutator(&join->rarg, context))
+ return true;
+ if (mutator(&join->quals, context))
+ return true;
+ if (mutator(&join->alias, context))
+ return true;
+ /* using list is deemed uninteresting */
+ }
+ break;
+ case T_IntoClause:
+ {
+ IntoClause *into = (IntoClause *) node;
+
+ if (mutator(&into->rel, context))
+ return true;
+ /* colNames, options are deemed uninteresting */
+ /* viewQuery should be null in raw parsetree, but check it */
+ if (mutator(&into->viewQuery, context))
+ return true;
+ }
+ break;
+ case T_List:
+ foreach(temp, (List *) node)
+ {
+ if (mutator(&lfirst(temp), context))
+ return true;
+ }
+ break;
+ case T_InsertStmt:
+ {
+ InsertStmt *stmt = (InsertStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->cols, context))
+ return true;
+ if (mutator(&stmt->selectStmt, context))
+ return true;
+ if (mutator(&stmt->onConflictClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_DeleteStmt:
+ {
+ DeleteStmt *stmt = (DeleteStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->usingClause, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_UpdateStmt:
+ {
+ UpdateStmt *stmt = (UpdateStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->fromClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_SelectStmt:
+ {
+ SelectStmt *stmt = (SelectStmt *) node;
+
+ if (mutator(&stmt->distinctClause, context))
+ return true;
+ if (mutator(&stmt->intoClause, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->fromClause, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->groupClause, context))
+ return true;
+ if (mutator(&stmt->havingClause, context))
+ return true;
+ if (mutator(&stmt->windowClause, context))
+ return true;
+ if (mutator(&stmt->valuesLists, context))
+ return true;
+ if (mutator(&stmt->sortClause, context))
+ return true;
+ if (mutator(&stmt->limitOffset, context))
+ return true;
+ if (mutator(&stmt->limitCount, context))
+ return true;
+ if (mutator(&stmt->lockingClause, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ if (mutator(&stmt->larg, context))
+ return true;
+ if (mutator(&stmt->rarg, context))
+ return true;
+ }
+ break;
+ case T_A_Expr:
+ {
+ A_Expr *expr = (A_Expr *) node;
+
+ if (mutator(&expr->lexpr, context))
+ return true;
+ if (mutator(&expr->rexpr, context))
+ return true;
+ /* operator name is deemed uninteresting */
+ }
+ break;
+ case T_BoolExpr:
+ {
+ BoolExpr *expr = (BoolExpr *) node;
+
+ if (mutator(&expr->args, context))
+ return true;
+ }
+ break;
+ case T_ColumnRef:
+ /* we assume the fields contain nothing interesting */
+ break;
+ case T_FuncCall:
+ {
+ FuncCall *fcall = (FuncCall *) node;
+
+ if (mutator(&fcall->args, context))
+ return true;
+ if (mutator(&fcall->agg_order, context))
+ return true;
+ if (mutator(&fcall->agg_filter, context))
+ return true;
+ if (mutator(&fcall->over, context))
+ return true;
+ /* function name is deemed uninteresting */
+ }
+ break;
+ case T_NamedArgExpr:
+ return mutator(&((NamedArgExpr *) node)->arg, context);
+ case T_A_Indices:
+ {
+ A_Indices *indices = (A_Indices *) node;
+
+ if (mutator(&indices->lidx, context))
+ return true;
+ if (mutator(&indices->uidx, context))
+ return true;
+ }
+ break;
+ case T_A_Indirection:
+ {
+ A_Indirection *indir = (A_Indirection *) node;
+
+ if (mutator(&indir->arg, context))
+ return true;
+ if (mutator(&indir->indirection, context))
+ return true;
+ }
+ break;
+ case T_A_ArrayExpr:
+ return mutator(&((A_ArrayExpr *) node)->elements, context);
+ case T_ResTarget:
+ {
+ ResTarget *rt = (ResTarget *) node;
+
+ if (mutator(&rt->indirection, context))
+ return true;
+ if (mutator(&rt->val, context))
+ return true;
+ }
+ break;
+ case T_MultiAssignRef:
+ return mutator(&((MultiAssignRef *) node)->source, context);
+ case T_TypeCast:
+ {
+ TypeCast *tc = (TypeCast *) node;
+
+ if (mutator(&tc->arg, context))
+ return true;
+ if (mutator(&tc->typeName, context))
+ return true;
+ }
+ break;
+ case T_CollateClause:
+ return mutator(&((CollateClause *) node)->arg, context);
+ case T_SortBy:
+ return mutator(&((SortBy *) node)->node, context);
+ case T_WindowDef:
+ {
+ WindowDef *wd = (WindowDef *) node;
+
+ if (mutator(&wd->partitionClause, context))
+ return true;
+ if (mutator(&wd->orderClause, context))
+ return true;
+ if (mutator(&wd->startOffset, context))
+ return true;
+ if (mutator(&wd->endOffset, context))
+ return true;
+ }
+ break;
+ case T_RangeSubselect:
+ {
+ RangeSubselect *rs = (RangeSubselect *) node;
+
+ if (mutator(&rs->subquery, context))
+ return true;
+ if (mutator(&rs->alias, context))
+ return true;
+ }
+ break;
+ case T_RangeFunction:
+ {
+ RangeFunction *rf = (RangeFunction *) node;
+
+ if (mutator(&rf->functions, context))
+ return true;
+ if (mutator(&rf->alias, context))
+ return true;
+ if (mutator(&rf->coldeflist, context))
+ return true;
+ }
+ break;
+ case T_RangeTableSample:
+ {
+ RangeTableSample *rts = (RangeTableSample *) node;
+
+ if (mutator(&rts->relation, context))
+ return true;
+ /* method name is deemed uninteresting */
+ if (mutator(&rts->args, context))
+ return true;
+ if (mutator(&rts->repeatable, context))
+ return true;
+ }
+ break;
+ case T_TypeName:
+ {
+ TypeName *tn = (TypeName *) node;
+
+ if (mutator(&tn->typmods, context))
+ return true;
+ if (mutator(&tn->arrayBounds, context))
+ return true;
+ /* type name itself is deemed uninteresting */
+ }
+ break;
+ case T_ColumnDef:
+ {
+ ColumnDef *coldef = (ColumnDef *) node;
+
+ if (mutator(&coldef->typeName, context))
+ return true;
+ if (mutator(&coldef->raw_default, context))
+ return true;
+ if (mutator(&coldef->collClause, context))
+ return true;
+ /* for now, constraints are ignored */
+ }
+ break;
+ case T_IndexElem:
+ {
+ IndexElem *indelem = (IndexElem *) node;
+
+ if (mutator(&indelem->expr, context))
+ return true;
+ /* collation and opclass names are deemed uninteresting */
+ }
+ break;
+ case T_GroupingSet:
+ return mutator(&((GroupingSet *) node)->content, context);
+ case T_LockingClause:
+ return mutator(&((LockingClause *) node)->lockedRels, context);
+ case T_XmlSerialize:
+ {
+ XmlSerialize *xs = (XmlSerialize *) node;
+
+ if (mutator(&xs->expr, context))
+ return true;
+ if (mutator(&xs->typeName, context))
+ return true;
+ }
+ break;
+ case T_WithClause:
+ return mutator(&((WithClause *) node)->ctes, context);
+ case T_InferClause:
+ {
+ InferClause *stmt = (InferClause *) node;
+
+ if (mutator(&stmt->indexElems, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ }
+ break;
+ case T_OnConflictClause:
+ {
+ OnConflictClause *stmt = (OnConflictClause *) node;
+
+ if (mutator(&stmt->infer, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ }
+ break;
+ case T_CommonTableExpr:
+ return mutator(&((CommonTableExpr *) node)->ctequery, context);
+ default:
+ return true;
+ }
+ return false;
+}
+
+/*
* planstate_tree_walker --- walk plan state trees
*
* The walker has already visited the current node, and so we need only
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3055b48..a36ac84 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -42,11 +42,13 @@
#include "catalog/pg_type.h"
#include "commands/async.h"
#include "commands/prepare.h"
+#include "commands/defrem.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "nodes/print.h"
+#include "nodes/nodeFuncs.h"
#include "optimizer/planner.h"
#include "pgstat.h"
#include "pg_trace.h"
@@ -73,6 +75,7 @@
#include "utils/snapmgr.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
+#include "utils/int8.h"
#include "mb/pg_wchar.h"
@@ -188,7 +191,8 @@ static bool IsTransactionStmtList(List *pstmts);
static void drop_unnamed_stmt(void);
static void SigHupHandler(SIGNAL_ARGS);
static void log_disconnections(int code, Datum arg);
-
+static bool exec_cached_query(const char* query, List *parsetree_list);
+static void exec_prepared_plan(Portal portal, const char *portal_name, long max_rows, CommandDest dest);
/* ----------------------------------------------------------------
* routines to obtain user input
@@ -958,6 +962,14 @@ exec_simple_query(const char *query_string)
isTopLevel = (list_length(parsetree_list) == 1);
/*
+ * Try to find cached plan
+ */
+ if (isTopLevel && autoprepare_threshold != 0 && exec_cached_query(query_string, parsetree_list))
+ {
+ return;
+ }
+
+ /*
* Run through the raw parsetree(s) and process each one.
*/
foreach(parsetree_item, parsetree_list)
@@ -1841,9 +1853,28 @@ exec_bind_message(StringInfo input_message)
static void
exec_execute_message(const char *portal_name, long max_rows)
{
- CommandDest dest;
+ Portal portal = GetPortalByName(portal_name);
+ CommandDest dest = whereToSendOutput;
+
+ /* Adjust destination to tell printtup.c what to do */
+ if (dest == DestRemote)
+ dest = DestRemoteExecute;
+
+ if (!PortalIsValid(portal))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_CURSOR),
+ errmsg("portal \"%s\" does not exist", portal_name)));
+
+ exec_prepared_plan(portal, portal_name, max_rows, dest);
+}
+
+/*
+ * Execute prepared plan.
+ */
+static void
+exec_prepared_plan(Portal portal, const char *portal_name, long max_rows, CommandDest dest)
+{
DestReceiver *receiver;
- Portal portal;
bool completed;
char completionTag[COMPLETION_TAG_BUFSIZE];
const char *sourceText;
@@ -1855,17 +1886,6 @@ exec_execute_message(const char *portal_name, long max_rows)
bool was_logged = false;
char msec_str[32];
- /* Adjust destination to tell printtup.c what to do */
- dest = whereToSendOutput;
- if (dest == DestRemote)
- dest = DestRemoteExecute;
-
- portal = GetPortalByName(portal_name);
- if (!PortalIsValid(portal))
- ereport(ERROR,
- (errcode(ERRCODE_UNDEFINED_CURSOR),
- errmsg("portal \"%s\" does not exist", portal_name)));
-
/*
* If the original query was a null string, just return
* EmptyQueryResponse.
@@ -1928,7 +1948,7 @@ exec_execute_message(const char *portal_name, long max_rows)
* context, because that may get deleted if portal contains VACUUM).
*/
receiver = CreateDestReceiver(dest);
- if (dest == DestRemoteExecute)
+ if (dest == DestRemoteExecute || dest == DestRemote)
SetRemoteDestReceiverParams(receiver, portal);
/*
@@ -4507,3 +4527,768 @@ log_disconnections(int code, Datum arg)
port->user_name, port->database_name, port->remote_host,
port->remote_port[0] ? " port=" : "", port->remote_port)));
}
+
+/*
+ * Autoprepare implementation.
+ * Autoprepare consists of raw parse tree mutator, hash table of cached plans and exec_cached_query function
+ * which combines exec_parse_message + exec_bind_message + exec_execute_message
+ */
+
+/*
+ * Mapping between parameters and replaced literals
+ */
+typedef struct ParamBinding
+{
+ A_Const* literal; /* Original literal */
+ ParamRef* paramref;/* Constructed parameter reference */
+ Param* param; /* Constructed parameter */
+ Node** ref; /* Pointer to pointer to literal node (used to revert raw parse tree update) */
+ Oid raw_type;/* Parameter raw type */
+ Oid type; /* Parameter type after analysis */
+ struct ParamBinding* next; /* L1-list of query parameter bindings */
+} ParamBinding;
+
+/*
+ * Plan cache entry
+ */
+typedef struct
+{
+ Node* parse_tree; /* tree is used as hash key */
+ dlist_node lru; /* double linked list to implement LRU */
+ int64 exec_count; /* counter of execution of this query */
+ CachedPlanSource* plan;
+ uint32 hash; /* hash calculated for this parsed tree */
+ Oid* param_types;/* types of parameters */
+ int n_params; /* number of parameters extracted for this query */
+ int16 format; /* portal output format */
+ bool disable_autoprepare; /* disable preparing of this query */
+} plan_cache_entry;
+
+static uint32 plan_cache_hash_fn(const void *key, Size keysize)
+{
+ return ((plan_cache_entry*)key)->hash;
+}
+
+static int plan_cache_match_fn(const void *key1, const void *key2, Size keysize)
+{
+ plan_cache_entry* e1 = (plan_cache_entry*)key1;
+ plan_cache_entry* e2 = (plan_cache_entry*)key2;
+
+ return equal(e1->parse_tree, e2->parse_tree)
+ && memcmp(e1->param_types, e2->param_types, sizeof(Oid)*e1->n_params) == 0 ? 0 : 1;
+}
+
+static void* plan_cache_keycopy_fn(void *dest, const void *src, Size keysize)
+{
+ plan_cache_entry* dst_entry = (plan_cache_entry*)dest;
+ plan_cache_entry* src_entry = (plan_cache_entry*)src;
+ dst_entry->parse_tree = copyObject(src_entry->parse_tree);
+ dst_entry->param_types = palloc(src_entry->n_params*sizeof(Oid));
+ dst_entry->n_params = src_entry->n_params;
+ memcpy(dst_entry->param_types, src_entry->param_types, src_entry->n_params*sizeof(Oid));
+ dst_entry->hash = src_entry->hash;
+ return dest;
+}
+
+#define PLAN_CACHE_SIZE 113
+
+/*
+ * Plan cache access statistic
+ */
+size_t autoprepare_hits;
+size_t autoprepare_misses;
+size_t autoprepare_cached_plans;
+
+/*
+ * Context for raw_expression_tree_mutator
+ */
+typedef struct {
+ int n_params; /* Number of extracted parameters */
+ uint32 hash; /* We calculate hash for parse tree during plan traversal */
+ ParamBinding** param_list_tail; /* pointer to last element "next" field address, used to construct L1 list of parameters */
+} GeneralizerCtx;
+
+/*
+ * Check if expression is constant (used to eliminate substitution of literals with parameters in such expressions
+ */
+static bool is_constant_expression(Node* node)
+{
+ return node != NULL
+ && (IsA(node, A_Const)
+ || (IsA(node, A_Expr)
+ && is_constant_expression(((A_Expr*)node)->lexpr)
+ && is_constant_expression(((A_Expr*)node)->rexpr)));
+}
+
+/*
+ * Infer type of literal expression. Null literals should not be replaced with parameters.
+ */
+static Oid get_literal_type(Value* val)
+{
+ int64 val64;
+ switch (val->type)
+ {
+ case T_Integer:
+ return INT4OID;
+ case T_Float:
+ /* could be an oversize integer as well as a float ... */
+ if (scanint8(strVal(val), true, &val64))
+ {
+ /*
+ * It might actually fit in int32. Probably only INT_MIN can
+ * occur, but we'll code the test generally just to be sure.
+ */
+ int32 val32 = (int32) val64;
+ return (val64 == (int64)val32) ? INT4OID : INT8OID;
+ }
+ else
+ {
+ return NUMERICOID;
+ }
+ case T_BitString:
+ return BITOID;
+ case T_String:
+ return UNKNOWNOID;
+ default:
+ Assert(false);
+ return InvalidOid;
+ }
+}
+
+static Datum get_param_value(Oid type, Value* val)
+{
+ if (val->type == T_Integer && type == INT4OID)
+ {
+ /*
+ * Integer constant
+ */
+ return Int32GetDatum((int32)val->val.ival);
+ }
+ else
+ {
+ /*
+ * Convert from string literal
+ */
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(type, &typinput, &typioparam);
+ return OidInputFunctionCall(typinput, val->val.str, typioparam, -1);
+ }
+}
+
+
+/*
+ * Callback for raw_expression_tree_mutator performing substitution of literals with parameters
+ */
+static bool
+raw_parse_tree_generalizer(Node** ref, void *context)
+{
+ Node* node = *ref;
+ GeneralizerCtx* ctx = (GeneralizerCtx*)context;
+ if (node == NULL)
+ {
+ return false;
+ }
+ /*
+ * Calculate hash for parse tree. We consider only node tags here, precise comparison of trees is done using equal() function.
+ * Here we calculate hash for original (unpatched) tree, without ParamRef nodes.
+ * It is non principle, because hash calculation doesn't take in account types and values of Const nodes. So the same generalized queries
+ * will have the same hash value. There are about 1000 different nodes tags, this is why we rotate hash on 10 bits.
+ */
+ ctx->hash = (ctx->hash << 10) ^ (ctx->hash >> 22) ^ nodeTag(node);
+
+ switch (nodeTag(node))
+ {
+ case T_A_Expr:
+ {
+ /*
+ * Do not perform substitution of literals in constant expression (which is likely to be the same for all queries and optimized by compiler)
+ */
+ if (!is_constant_expression(node))
+ {
+ A_Expr *expr = (A_Expr *) node;
+ if (raw_parse_tree_generalizer((Node**)&expr->lexpr, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&expr->rexpr, context))
+ return true;
+ }
+ break;
+ }
+ case T_A_Const:
+ {
+ /*
+ * Do substitution of literals with parameters here
+ */
+ A_Const* literal = (A_Const*)node;
+ if (literal->val.type != T_Null)
+ {
+ /*
+ * Do not substitute null literals with parameters
+ */
+ ParamBinding* cp = palloc0(sizeof(ParamBinding));
+ ParamRef* param = makeNode(ParamRef);
+ param->number = ++ctx->n_params;
+ param->location = literal->location;
+ cp->ref = ref;
+ cp->paramref = param;
+ cp->literal = literal;
+ cp->raw_type = get_literal_type(&literal->val);
+ *ctx->param_list_tail = cp;
+ ctx->param_list_tail = &cp->next;
+ *ref = (Node*)param;
+ }
+ break;
+ }
+ case T_SelectStmt:
+ {
+ /*
+ * Substitute literals only in target list, WHERE, VALUES and WITH clause,
+ * skipping target and from lists, which is unlikely contains some parameterized values
+ */
+ SelectStmt *stmt = (SelectStmt *) node;
+ if (stmt->intoClause)
+ return true; /* Utility statement can not be prepared */
+ if (raw_parse_tree_generalizer((Node**)&stmt->targetList, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->whereClause, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->valuesLists, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->withClause, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->larg, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->rarg, context))
+ return true;
+ break;
+ }
+ case T_TypeName:
+ case T_SortGroupClause:
+ case T_SortBy:
+ case T_A_ArrayExpr:
+ case T_TypeCast:
+ /*
+ * Literals in this clauses should not be replaced with parameters
+ */
+ break;
+ default:
+ /*
+ * Default traversal. raw_expression_tree_mutator returns true for all not recognized nodes, for example right now
+ * all transaction control statements are not covered by raw_expression_tree_mutator and so will not auto prepared.
+ * My experiments show that effect of non-preparing start/commit transaction statements is positive.
+ */
+ return raw_expression_tree_mutator(node, raw_parse_tree_generalizer, context);
+ }
+ return false;
+}
+
+static Node*
+parse_tree_generalizer(Node *node, void *context)
+{
+ ParamBinding* binding;
+ ParamBinding* binding_list = (ParamBinding*)context;
+ if (node == NULL)
+ {
+ return NULL;
+ }
+ if (IsA(node, Query))
+ {
+ return (Node*)query_tree_mutator((Query*)node,
+ parse_tree_generalizer,
+ context,
+ QTW_DONT_COPY_QUERY);
+ }
+ if (IsA(node, Const))
+ {
+ Const* c = (Const*)node;
+ int paramno = 1;
+ for (binding = binding_list; binding != NULL && binding->literal->location != c->location; binding = binding->next, paramno++);
+ if (binding != NULL)
+ {
+ if (binding->param != NULL)
+ {
+ return (Node*)binding->param;
+ }
+ else
+ {
+ Param* param = makeNode(Param);
+ param->paramkind = PARAM_EXTERN;
+ param->paramid = paramno;
+ param->paramtype = c->consttype;
+ param->paramtypmod = c->consttypmod;
+ param->paramcollid = c->constcollid;
+ param->location = c->location;
+ binding->type = c->consttype;
+ binding->param = param;
+ return (Node*)param;
+ }
+ }
+ return node;
+ }
+ return expression_tree_mutator(node, parse_tree_generalizer, context);
+}
+
+/*
+ * Restore original parse tree, replacing all ParamRef back with Const nodes.
+ * Such undo operation seems to be more efficient than copying the whole parse tree by raw_expression_tree_mutator
+ */
+static void undo_query_plan_changes(ParamBinding* cp)
+{
+ while (cp != NULL) {
+ *cp->ref = (Node*)cp->literal;
+ cp = cp->next;
+ }
+}
+
+/*
+ * Callback for raw_expression_tree_walker dropping parse tree
+ */
+static bool drop_tree_node(Node* node, void* context)
+{
+ if (node) {
+ raw_expression_tree_walker(node, drop_tree_node, NULL);
+ pfree(node);
+ }
+ return false;
+}
+
+/*
+ * Location of converted literal in query.
+ * Used for precise error reporting (line number)
+ */
+static int param_location;
+
+/*
+ * Error callback adding information about error location
+ */
+static void
+prepare_error_callback(void *arg)
+{
+ CachedPlanSource *psrc = (CachedPlanSource*)arg;
+ /* And pass it to the ereport mechanism */
+ if (geterrcode() != ERRCODE_QUERY_CANCELED) {
+ int pos = pg_mbstrlen_with_len(psrc->query_string, param_location) + 1;
+ (void)errposition(pos);
+ }
+}
+
+
+/*
+ * Try to generalize query, find cached plan for it and execute
+ */
+static bool exec_cached_query(const char *query_string, List *parsetree_list)
+{
+ int n_params;
+ plan_cache_entry *entry;
+ bool found;
+ MemoryContext old_context;
+ CachedPlanSource *psrc;
+ ParamListInfo params;
+ int paramno;
+ CachedPlan *cplan;
+ Portal portal;
+ bool snapshot_set = false;
+ GeneralizerCtx ctx;
+ ParamBinding* binding;
+ ParamBinding* binding_list;
+ plan_cache_entry pattern;
+ Oid* param_types;
+ RawStmt *raw_parse_tree;
+
+ static HTAB* plan_cache; /* hash table for plan cache */
+ static dlist_head lru; /* LRU L2-list for cached queries */
+ static MemoryContext plan_cache_context; /* memory context used for plan cache */
+
+ raw_parse_tree = castNode(RawStmt, linitial(parsetree_list));
+
+ /*
+ * Substitute literals with parameters and calculate hash for raw parse tree
+ */
+ ctx.param_list_tail = &binding_list;
+ ctx.n_params = 0;
+ ctx.hash = 0;
+ if (raw_parse_tree_generalizer(&raw_parse_tree->stmt, &ctx))
+ {
+ *ctx.param_list_tail = NULL;
+ undo_query_plan_changes(binding_list);
+ autoprepare_misses += 1;
+ return false;
+ }
+ *ctx.param_list_tail = NULL;
+ n_params = ctx.n_params;
+
+ /*
+ * Extract array of parameters types: it is needed for cached plan lookup
+ */
+ param_types = (Oid*)palloc(sizeof(Oid)*n_params);
+ for (paramno = 0, binding = binding_list; paramno < n_params; paramno++, binding = binding->next)
+ {
+ param_types[paramno] = binding->raw_type;
+ }
+
+ /*
+ * Construct plan cache context if not constructed yet.
+ */
+ if (plan_cache_context == NULL)
+ {
+ plan_cache_context = AllocSetContextCreate(TopMemoryContext,
+ "plan cache context",
+ ALLOCSET_DEFAULT_SIZES);
+ }
+ /* Manipulations with hash table are performed in plan_cache_context memory context */
+ old_context = MemoryContextSwitchTo(plan_cache_context);
+
+ /*
+ * Initialize hash table if not initialized yet
+ */
+ if (plan_cache == NULL)
+ {
+ static HASHCTL info;
+ info.keysize = sizeof(plan_cache_entry);
+ info.entrysize = sizeof(plan_cache_entry);
+ info.hash = plan_cache_hash_fn;
+ info.match = plan_cache_match_fn;
+ info.keycopy = plan_cache_keycopy_fn;
+ plan_cache = hash_create("plan_cache", autoprepare_limit != 0 ? autoprepare_limit : PLAN_CACHE_SIZE,
+ &info, HASH_ELEM | HASH_FUNCTION | HASH_COMPARE | HASH_KEYCOPY);
+ dlist_init(&lru);
+ }
+
+ /*
+ * Lookup generalized query
+ */
+ pattern.parse_tree = raw_parse_tree->stmt;
+ pattern.hash = ctx.hash;
+ pattern.n_params = n_params;
+ pattern.param_types = param_types;
+ entry = (plan_cache_entry*)hash_search(plan_cache, &pattern, HASH_ENTER, &found);
+ if (!found)
+ {
+ /* Check number of cached queries */
+ if (++autoprepare_cached_plans > autoprepare_limit && autoprepare_limit != 0)
+ {
+ /* Drop least recently accessed query */
+ plan_cache_entry* victim = dlist_container(plan_cache_entry, lru, lru.head.prev);
+ Node* dropped_tree = victim->parse_tree;
+ dlist_delete(&victim->lru);
+ if (victim->plan)
+ {
+ DropCachedPlan(victim->plan);
+ }
+ pfree(victim->param_types);
+ hash_search(plan_cache, victim, HASH_REMOVE, NULL);
+ raw_expression_tree_walker(dropped_tree, drop_tree_node, NULL);
+ autoprepare_cached_plans -= 1;
+ }
+ entry->exec_count = 0;
+ entry->plan = NULL;
+ entry->disable_autoprepare = false;
+ }
+ else
+ {
+ dlist_delete(&entry->lru); /* accessed entry will be moved to the head of LRU list */
+ if (entry->plan != NULL && !entry->plan->is_valid)
+ {
+ /* Drop invalidated plan: it will be reconstructed later */
+ DropCachedPlan(entry->plan);
+ entry->plan = NULL;
+ }
+ }
+ dlist_insert_after(&lru.head, &entry->lru); /* prepend entry to the head of LRU list */
+ MemoryContextSwitchTo(old_context); /* Done with plan_cache_context memory context */
+
+ /*
+ * Prepare query only when it is executed more than autoprepare_threshold times
+ */
+ if (entry->disable_autoprepare || entry->exec_count++ < autoprepare_threshold)
+ {
+ undo_query_plan_changes(binding_list);
+ autoprepare_misses += 1;
+ return false;
+ }
+ if (entry->plan == NULL)
+ {
+ bool snapshot_set = false;
+ const char *commandTag;
+ List *querytree_list;
+ Query *query;
+
+ /*
+ * Switch to appropriate context for preparing plan.
+ */
+ old_context = MemoryContextSwitchTo(MessageContext);
+
+ /*
+ * Get the command name for use in status display (it also becomes the
+ * default completion tag, down inside PortalRun). Set ps_status and
+ * do any special start-of-SQL-command processing needed by the
+ * destination.
+ */
+ commandTag = CreateCommandTag(raw_parse_tree->stmt);
+
+ /*
+ * If we are in an aborted transaction, reject all commands except
+ * COMMIT/ABORT. It is important that this test occur before we try
+ * to do parse analysis, rewrite, or planning, since all those phases
+ * try to do database accesses, which may fail in abort state. (It
+ * might be safe to allow some additional utility commands in this
+ * state, but not many...)
+ */
+ if (IsAbortedTransactionBlockState() &&
+ !IsTransactionExitStmt(raw_parse_tree->stmt))
+ ereport(ERROR,
+ (errcode(ERRCODE_IN_FAILED_SQL_TRANSACTION),
+ errmsg("current transaction is aborted, "
+ "commands ignored until end of transaction block"),
+ errdetail_abort()));
+
+ /*
+ * Create the CachedPlanSource before we do parse analysis, since it
+ * needs to see the unmodified raw parse tree.
+ */
+ psrc = CreateCachedPlan(raw_parse_tree, query_string, commandTag);
+
+ /*
+ * Revert raw plan to use literals
+ */
+ undo_query_plan_changes(binding_list);
+
+ /*
+ * Set up a snapshot if parse analysis/planning will need one.
+ */
+ if (analyze_requires_snapshot(raw_parse_tree))
+ {
+ PushActiveSnapshot(GetTransactionSnapshot());
+ snapshot_set = true;
+ }
+
+ querytree_list = pg_analyze_and_rewrite(raw_parse_tree, query_string,
+ NULL, 0);
+ /*
+ * Replace Const with Param nodes
+ */
+ (void)query_tree_mutator((Query*)linitial(querytree_list),
+ parse_tree_generalizer,
+ binding_list,
+ QTW_DONT_COPY_QUERY);
+
+ /* Done with the snapshot used for parsing/planning */
+ if (snapshot_set)
+ PopActiveSnapshot();
+
+ param_types = (Oid*)palloc(sizeof(Oid)*n_params);
+ psrc->param_types = param_types;
+ for (paramno = 0, binding = binding_list; paramno < n_params; paramno++, binding = binding->next)
+ {
+ if (binding->param == NULL)
+ {
+ /* Failed to resolve parameter type */
+ entry->disable_autoprepare = true;
+ autoprepare_misses += 1;
+ MemoryContextSwitchTo(old_context);
+ return false;
+ }
+ param_types[paramno] = binding->type;
+ }
+
+ /* Finish filling in the CachedPlanSource */
+ CompleteCachedPlan(psrc,
+ querytree_list,
+ NULL,
+ param_types,
+ n_params,
+ NULL,
+ NULL,
+ CURSOR_OPT_PARALLEL_OK, /* allow parallel mode */
+ true); /* fixed result */
+
+ /* If we got a cancel signal during analysis, quit */
+ CHECK_FOR_INTERRUPTS();
+
+ SaveCachedPlan(psrc);
+
+ /*
+ * We do NOT close the open transaction command here; that only happens
+ * when the client sends Sync. Instead, do CommandCounterIncrement just
+ * in case something happened during parse/plan.
+ */
+ CommandCounterIncrement();
+
+ MemoryContextSwitchTo(old_context); /* Done with MessageContext memory context */
+
+ entry->plan = psrc;
+
+ /*
+ * Determine output format
+ */
+ entry->format = 0; /* TEXT is default */
+ if (IsA(raw_parse_tree->stmt, FetchStmt))
+ {
+ FetchStmt *stmt = (FetchStmt *)raw_parse_tree->stmt;
+
+ if (!stmt->ismove)
+ {
+ Portal fportal = GetPortalByName(stmt->portalname);
+
+ if (PortalIsValid(fportal) &&
+ (fportal->cursorOptions & CURSOR_OPT_BINARY))
+ entry->format = 1; /* BINARY */
+ }
+ }
+ }
+ else
+ {
+ /* Plan found */
+ psrc = entry->plan;
+ Assert(n_params == entry->n_params);
+ }
+
+ /*
+ * If we are in aborted transaction state, the only portals we can
+ * actually run are those containing COMMIT or ROLLBACK commands. We
+ * disallow binding anything else to avoid problems with infrastructure
+ * that expects to run inside a valid transaction. We also disallow
+ * binding any parameters, since we can't risk calling user-defined I/O
+ * functions.
+ */
+ if (IsAbortedTransactionBlockState() &&
+ (!IsTransactionExitStmt(psrc->raw_parse_tree->stmt) ||
+ n_params != 0))
+ ereport(ERROR,
+ (errcode(ERRCODE_IN_FAILED_SQL_TRANSACTION),
+ errmsg("current transaction is aborted, "
+ "commands ignored until end of transaction block"),
+ errdetail_abort()));
+
+ /*
+ * Create unnamed portal to run the query or queries in. If there
+ * already is one, silently drop it.
+ */
+ portal = CreatePortal("", true, true);
+ /* Don't display the portal in pg_cursors */
+ portal->visible = false;
+
+ /*
+ * Prepare to copy stuff into the portal's memory context. We do all this
+ * copying first, because it could possibly fail (out-of-memory) and we
+ * don't want a failure to occur between GetCachedPlan and
+ * PortalDefineQuery; that would result in leaking our plancache refcount.
+ */
+ old_context = MemoryContextSwitchTo(PortalGetHeapMemory(portal));
+
+ /* Copy the plan's query string into the portal */
+ query_string = pstrdup(psrc->query_string);
+
+ /*
+ * Set a snapshot if we have parameters to fetch (since the input
+ * functions might need it) or the query isn't a utility command (and
+ * hence could require redoing parse analysis and planning). We keep the
+ * snapshot active till we're done, so that plancache.c doesn't have to
+ * take new ones.
+ */
+ if (n_params > 0 ||
+ (psrc->raw_parse_tree &&
+ analyze_requires_snapshot(psrc->raw_parse_tree)))
+ {
+ PushActiveSnapshot(GetTransactionSnapshot());
+ snapshot_set = true;
+ }
+
+ /*
+ * Fetch parameters, if any, and store in the portal's memory context.
+ */
+ if (n_params > 0)
+ {
+ ErrorContextCallback errcallback;
+
+ params = (ParamListInfo) palloc0(offsetof(ParamListInfoData, params) +
+ n_params * sizeof(ParamExternData));
+ params->numParams = n_params;
+
+ /*
+ * Register error callback to precisely report error in case of conversion error while storig parameter value.
+ */
+ errcallback.callback = prepare_error_callback;
+ errcallback.arg = (void *) psrc;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ for (paramno = 0, binding = binding_list;
+ paramno < n_params;
+ paramno++, binding = binding->next)
+ {
+ Oid ptype = psrc->param_types[paramno];
+
+ param_location = binding->literal->location;
+
+ params->params[paramno].isnull = false;
+ params->params[paramno].value = get_param_value(ptype, &binding->literal->val);
+ /*
+ * We mark the params as CONST. This ensures that any custom plan
+ * makes full use of the parameter values.
+ */
+ params->params[paramno].pflags = PARAM_FLAG_CONST;
+ params->params[paramno].ptype = ptype;
+ }
+ error_context_stack = errcallback.previous;
+ }
+ else
+ {
+ params = NULL;
+ }
+
+ /* Done storing stuff in portal's context */
+ MemoryContextSwitchTo(old_context);
+
+ /*
+ * Obtain a plan from the CachedPlanSource. Any cruft from (re)planning
+ * will be generated in MessageContext. The plan refcount will be
+ * assigned to the Portal, so it will be released at portal destruction.
+ */
+ cplan = GetCachedPlan(psrc, params, false);
+
+ /*
+ * Now we can define the portal.
+ *
+ * DO NOT put any code that could possibly throw an error between the
+ * above GetCachedPlan call and here.
+ */
+ PortalDefineQuery(portal,
+ NULL,
+ query_string,
+ psrc->commandTag,
+ cplan->stmt_list,
+ cplan);
+
+ /* Done with the snapshot used for parameter I/O and parsing/planning */
+ if (snapshot_set)
+ {
+ PopActiveSnapshot();
+ }
+
+ /*
+ * And we're ready to start portal execution.
+ */
+ PortalStart(portal, params, 0, InvalidSnapshot);
+
+ /*
+ * Apply the result format requests to the portal.
+ */
+ PortalSetResultFormat(portal, 1, &entry->format);
+
+ /*
+ * Finally execute prepared statement
+ */
+ exec_prepared_plan(portal, "", FETCH_ALL, whereToSendOutput);
+
+ /*
+ * Close down transaction statement, if one is open.
+ */
+ finish_xact_command();
+
+ autoprepare_hits += 1;
+
+ return true;
+}
+
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 8b5f064..5ab83f0 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -476,6 +476,10 @@ int tcp_keepalives_idle;
int tcp_keepalives_interval;
int tcp_keepalives_count;
+
+int autoprepare_threshold;
+int autoprepare_limit;
+
/*
* SSL renegotiation was been removed in PostgreSQL 9.5, but we tolerate it
* being set to zero (meaning never renegotiate) for backward compatibility.
@@ -1960,6 +1964,28 @@ static struct config_int ConfigureNamesInt[] =
check_max_stack_depth, assign_max_stack_depth, NULL
},
+ /*
+ * Threshold for implicit preparing of frequently executed queries
+ */
+ {
+ {"autoprepare_threshold", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Threshold for autopreparing query."),
+ gettext_noop("0 value disables autoprepare.")
+ },
+ &autoprepare_threshold,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+ {
+ {"autoprepare_limit", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Maximal number of autoprepared queries."),
+ gettext_noop("0 means unlimited number of autoprepared queries. Too large number of prepared queries can cause backend memory overflow and slowdown execution speed (because of increased lookup time)")
+ },
+ &autoprepare_limit,
+ 113, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"temp_file_limit", PGC_SUSET, RESOURCES_DISK,
gettext_noop("Limits the total size of all temporary files used by each process."),
diff --git a/src/include/nodes/nodeFuncs.h b/src/include/nodes/nodeFuncs.h
index b6c9b48..1388822 100644
--- a/src/include/nodes/nodeFuncs.h
+++ b/src/include/nodes/nodeFuncs.h
@@ -73,8 +73,11 @@ extern Node *query_or_expression_tree_mutator(Node *node, Node *(*mutator) (),
extern bool raw_expression_tree_walker(Node *node, bool (*walker) (),
void *context);
+extern bool raw_expression_tree_mutator(Node *node, bool (*mutator) (),
+ void *context);
struct PlanState;
extern bool planstate_tree_walker(struct PlanState *planstate, bool (*walker) (),
void *context);
+
#endif /* NODEFUNCS_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 7dd3780..9cd0917 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -251,6 +251,9 @@ extern int client_min_messages;
extern int log_min_duration_statement;
extern int log_temp_files;
+extern int autoprepare_threshold;
+extern int autoprepare_limit;
+
extern int temp_file_limit;
extern int num_temp_buffers;
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-11 15:12 Bruce Momjian <[email protected]>
parent: Konstantin Knizhnik <[email protected]>
2 siblings, 2 replies; 51+ messages in thread
From: Bruce Momjian @ 2017-05-11 15:12 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On Wed, May 10, 2017 at 07:11:07PM +0300, Konstantin Knizhnik wrote:
> I am going to continue work on this patch I will be glad to receive any
> feedback and suggestions for its improvement.
> In most cases, applications are not accessing Postgres directly, but using
> some connection pooling layer and so them are not able to use prepared
> statements.
> But at simple OLTP Postgres spent more time on building query plan than on
> execution itself. And it is possible to speedup Postgres about two times at
> such workload!
> Another alternative is true shared plan cache. May be it is even more
> perspective approach, but definitely much more invasive and harder to
> implement.
Can we back up and get an overview of what you are doing and how you are
doing it? Our TODO list suggests this order for successful patches:
Desirability -> Design -> Implement -> Test -> Review -> Commit
You kind of started at the Implementation/patch level, which makes it
hard to evaluate.
I think everyone agrees on the Desirability of the feature, but the
Design is the tricky part. I think the design questions are:
* What information is stored about cached plans?
* How are the cached plans invalidated?
* How is a query matched against a cached plan?
Looking at the options, ideally the plan would be cached at the same
query stage as the stage where the incoming query is checked against the
cache. However, caching and checking at the same level offers no
benefit, so they are going to be different. For example, caching a
parse tree at the time it is created, then checking at the same point if
the incoming query is the same doesn't help you because you already had
to create the parse tree get to that point.
A more concrete example is prepared statements. They are stored at the
end of planning and matched in the parser. However, you can easily do
that since the incoming query specifies the name of the prepared query,
so there is no trick to matching.
The desire is to cache as late as possible so you cache more work and
you have more detail about the referenced objects, which helps with
cache invalidation. However, you also want to do cache matching as
early as possible to improve performance.
So, let's look at some options. One interesting idea from Doug Doole
was to do it between the tokenizer and parser. I think they are glued
together so you would need a way to run the tokenizer separately and
compare that to the tokens you stored for the cached plan. The larger
issue is that prepared plans already are checked after parsing, and we
know they are a win, so matching any earlier than that just seems like
overkill and likely to lead to lots of problems.
So, you could do it after parsing but before parse-analysis, which is
kind of what prepared queries do. One tricky problem is that we don't
bind the query string tokens to database objects until after parse
analysis.
Doing matching before parse-analysis is going to be tricky, which is why
there are so many comments about the approach. Changing search_path can
certainly affect it, but creating objects in earlier-mentioned schemas
can also change how an object reference in a query is resolved. Even
obscure things like the creation of a new operator that has higher
precedence in the query could change the plan, though am not sure if
our prepared query system even handles that properly.
Anyway, that is my feedback. I would like to get an overview of what
you are trying to do and the costs/benefits of each option so we can
best guide you.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-11 17:39 Douglas Doole <[email protected]>
parent: Bruce Momjian <[email protected]>
1 sibling, 1 reply; 51+ messages in thread
From: Douglas Doole @ 2017-05-11 17:39 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; Konstantin Knizhnik <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
>
> One interesting idea from Doug Doole was to do it between the tokenizer
> and parser. I think they are glued together so you would need a way to run
> the tokenizer separately and compare that to the tokens you stored for the
> cached plan.
>
When I did this, we had the same problem that the tokenizer and parser were
tightly coupled. Fortunately, I was able to do as you suggest and run the
tokenizer separately to do my analysis.
So my model was to do statement generalization before entering the compiler
at all. I would tokenize the statement to find the literals and generate a
new statement string with placeholders. The new string would the be passed
to the compiler which would then tokenize and parse the reworked statement.
This means we incurred the cost of tokenizing twice, but the tokenizer was
lightweight enough that it wasn't a problem. In exchange I was able to do
statement generalization without touching the compiler - the compiler saw
the generalized statement text as any other statement and handled it in the
exact same way. (There was just a bit of new code around variable binding.)
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-11 18:15 Bruce Momjian <[email protected]>
parent: Douglas Doole <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Bruce Momjian @ 2017-05-11 18:15 UTC (permalink / raw)
To: Douglas Doole <[email protected]>; +Cc: Konstantin Knizhnik <[email protected]>; Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On Thu, May 11, 2017 at 05:39:58PM +0000, Douglas Doole wrote:
> One interesting idea from Doug Doole was to do it between the tokenizer and
> parser. I think they are glued together so you would need a way to run the
> tokenizer separately and compare that to the tokens you stored for the
> cached plan.
>
>
> When I did this, we had the same problem that the tokenizer and parser were
> tightly coupled. Fortunately, I was able to do as you suggest and run the
> tokenizer separately to do my analysis.Â
>
> So my model was to do statement generalization before entering the compiler at
> all. I would tokenize the statement to find the literals and generate a new
> statement string with placeholders. The new string would the be passed to the
> compiler which would then tokenize and parse the reworked statement.
>
> This means we incurred the cost of tokenizing twice, but the tokenizer was
> lightweight enough that it wasn't a problem. In exchange I was able to do
> statement generalization without touching the compiler - the compiler saw the
> generalized statement text as any other statement and handled it in the exact
> same way. (There was just a bit of new code around variable binding.)
Good point. I think we need to do some measurements to see if the
parser-only stage is actually significant. I have a hunch that
commercial databases have much heavier parsers than we do.
This split would also not work if the scanner feeds changes back into
the parser. I know C does that for typedefs but I don't think we do.
Ideally I would like to see percentage-of-execution numbers for typical
queries for scan, parse, parse-analysis, plan, and execute to see where
the wins are.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-11 18:31 Tom Lane <[email protected]>
parent: Bruce Momjian <[email protected]>
0 siblings, 2 replies; 51+ messages in thread
From: Tom Lane @ 2017-05-11 18:31 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Douglas Doole <[email protected]>; Konstantin Knizhnik <[email protected]>; Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
Bruce Momjian <[email protected]> writes:
> Good point. I think we need to do some measurements to see if the
> parser-only stage is actually significant. I have a hunch that
> commercial databases have much heavier parsers than we do.
FWIW, gram.y does show up as significant in many of the profiles I take.
I speculate that this is not so much that it eats many CPU cycles, as that
the constant tables are so large as to incur lots of cache misses. scan.l
is not quite as big a deal for some reason, even though it's also large.
regards, tom lane
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-11 18:32 Andres Freund <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 0 replies; 51+ messages in thread
From: Andres Freund @ 2017-05-11 18:32 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Bruce Momjian <[email protected]>; +Cc: Douglas Doole <[email protected]>; Konstantin Knizhnik <[email protected]>; Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers
On May 11, 2017 11:31:02 AM PDT, Tom Lane <[email protected]> wrote:
>Bruce Momjian <[email protected]> writes:
>> Good point. I think we need to do some measurements to see if the
>> parser-only stage is actually significant. I have a hunch that
>> commercial databases have much heavier parsers than we do.
>
>FWIW, gram.y does show up as significant in many of the profiles I
>take.
>I speculate that this is not so much that it eats many CPU cycles, as
>that
>the constant tables are so large as to incur lots of cache misses.
>scan.l
>is not quite as big a deal for some reason, even though it's also
>large.
And that there's a lot of unpredictable branches, leading to a lot if pipeline stalls.
Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-11 19:41 Konstantin Knizhnik <[email protected]>
parent: Bruce Momjian <[email protected]>
1 sibling, 1 reply; 51+ messages in thread
From: Konstantin Knizhnik @ 2017-05-11 19:41 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On 05/11/2017 06:12 PM, Bruce Momjian wrote:
> On Wed, May 10, 2017 at 07:11:07PM +0300, Konstantin Knizhnik wrote:
>> I am going to continue work on this patch I will be glad to receive any
>> feedback and suggestions for its improvement.
>> In most cases, applications are not accessing Postgres directly, but using
>> some connection pooling layer and so them are not able to use prepared
>> statements.
>> But at simple OLTP Postgres spent more time on building query plan than on
>> execution itself. And it is possible to speedup Postgres about two times at
>> such workload!
>> Another alternative is true shared plan cache. May be it is even more
>> perspective approach, but definitely much more invasive and harder to
>> implement.
> Can we back up and get an overview of what you are doing and how you are
> doing it? Our TODO list suggests this order for successful patches:
>
> Desirability -> Design -> Implement -> Test -> Review -> Commit
>
> You kind of started at the Implementation/patch level, which makes it
> hard to evaluate.
>
> I think everyone agrees on the Desirability of the feature, but the
> Design is the tricky part. I think the design questions are:
>
> * What information is stored about cached plans?
> * How are the cached plans invalidated?
> * How is a query matched against a cached plan?
>
> Looking at the options, ideally the plan would be cached at the same
> query stage as the stage where the incoming query is checked against the
> cache. However, caching and checking at the same level offers no
> benefit, so they are going to be different. For example, caching a
> parse tree at the time it is created, then checking at the same point if
> the incoming query is the same doesn't help you because you already had
> to create the parse tree get to that point.
>
> A more concrete example is prepared statements. They are stored at the
> end of planning and matched in the parser. However, you can easily do
> that since the incoming query specifies the name of the prepared query,
> so there is no trick to matching.
>
> The desire is to cache as late as possible so you cache more work and
> you have more detail about the referenced objects, which helps with
> cache invalidation. However, you also want to do cache matching as
> early as possible to improve performance.
>
> So, let's look at some options. One interesting idea from Doug Doole
> was to do it between the tokenizer and parser. I think they are glued
> together so you would need a way to run the tokenizer separately and
> compare that to the tokens you stored for the cached plan. The larger
> issue is that prepared plans already are checked after parsing, and we
> know they are a win, so matching any earlier than that just seems like
> overkill and likely to lead to lots of problems.
>
> So, you could do it after parsing but before parse-analysis, which is
> kind of what prepared queries do. One tricky problem is that we don't
> bind the query string tokens to database objects until after parse
> analysis.
>
> Doing matching before parse-analysis is going to be tricky, which is why
> there are so many comments about the approach. Changing search_path can
> certainly affect it, but creating objects in earlier-mentioned schemas
> can also change how an object reference in a query is resolved. Even
> obscure things like the creation of a new operator that has higher
> precedence in the query could change the plan, though am not sure if
> our prepared query system even handles that properly.
>
> Anyway, that is my feedback. I would like to get an overview of what
> you are trying to do and the costs/benefits of each option so we can
> best guide you.
>
Sorry, for luck of overview.
I have started with small prototype just to investigate if such optimization makes sense or not.
When I get more than two time advantage in performance on standard pgbench, I come to conclusion that this
optimization can be really very useful and now try to find the best way of its implementation.
I have started with simplest approach when string literals are replaced with parameters. It is done before parsing.
And can be done very fast - just need to locate data in quotes.
But this approach is not safe and universal: you will not be able to speedup most of the existed queries without rewriting them.
This is why I have provided second implementation which replace literals with parameters after raw parsing.
Certainly it is slower than first approach. But still provide significant advantage in performance: more than two times at pgbench.
Then I tried to run regression tests and find several situations where type analysis is not correctly performed in case of replacing literals with parameters.
So my third attempt is to replace constant nodes with parameters in already analyzed tree.
Now answering your questions:
* What information is stored about cached plans?
Key to locate cached plan is raw parse tree. Value is saved CachedPlanSource.
* How are the cached plans invalidated?
In the same way as plans for explicitly prepared statements.
* How is a query matched against a cached plan?
Hash function is calculated for raw parse tree and equal() function is used to compare raw plans with literal nodes replaced with parameters.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-11 19:48 Konstantin Knizhnik <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 51+ messages in thread
From: Konstantin Knizhnik @ 2017-05-11 19:48 UTC (permalink / raw)
To: Tom Lane <[email protected]>; Bruce Momjian <[email protected]>; +Cc: Douglas Doole <[email protected]>; Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On 05/11/2017 09:31 PM, Tom Lane wrote:
> Bruce Momjian <[email protected]> writes:
>> Good point. I think we need to do some measurements to see if the
>> parser-only stage is actually significant. I have a hunch that
>> commercial databases have much heavier parsers than we do.
> FWIW, gram.y does show up as significant in many of the profiles I take.
> I speculate that this is not so much that it eats many CPU cycles, as that
> the constant tables are so large as to incur lots of cache misses. scan.l
> is not quite as big a deal for some reason, even though it's also large.
>
> regards, tom lane
Yes, my results shows that pg_parse_query adds not so much overhead:
206k TPS for my first variant with string literal substitution and modified query text used as hash key vs.
181k. TPS for version with patching raw parse tree constructed by pg_parse_query.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-11 19:52 Andres Freund <[email protected]>
parent: Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Andres Freund @ 2017-05-11 19:52 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Tom Lane <[email protected]>; Bruce Momjian <[email protected]>; Douglas Doole <[email protected]>; Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers
On 2017-05-11 22:48:26 +0300, Konstantin Knizhnik wrote:
> On 05/11/2017 09:31 PM, Tom Lane wrote:
> > Bruce Momjian <[email protected]> writes:
> > > Good point. I think we need to do some measurements to see if the
> > > parser-only stage is actually significant. I have a hunch that
> > > commercial databases have much heavier parsers than we do.
> > FWIW, gram.y does show up as significant in many of the profiles I take.
> > I speculate that this is not so much that it eats many CPU cycles, as that
> > the constant tables are so large as to incur lots of cache misses. scan.l
> > is not quite as big a deal for some reason, even though it's also large.
> >
> > regards, tom lane
> Yes, my results shows that pg_parse_query adds not so much overhead:
> 206k TPS for my first variant with string literal substitution and modified query text used as hash key vs.
> 181k. TPS for version with patching raw parse tree constructed by pg_parse_query.
Those numbers and your statement seem to contradict each other?
- Andres
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-11 20:30 Konstantin Knizhnik <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Konstantin Knizhnik @ 2017-05-11 20:30 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; Bruce Momjian <[email protected]>; Douglas Doole <[email protected]>; Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers
On 05/11/2017 10:52 PM, Andres Freund wrote:
> On 2017-05-11 22:48:26 +0300, Konstantin Knizhnik wrote:
>> On 05/11/2017 09:31 PM, Tom Lane wrote:
>>> Bruce Momjian <[email protected]> writes:
>>>> Good point. I think we need to do some measurements to see if the
>>>> parser-only stage is actually significant. I have a hunch that
>>>> commercial databases have much heavier parsers than we do.
>>> FWIW, gram.y does show up as significant in many of the profiles I take.
>>> I speculate that this is not so much that it eats many CPU cycles, as that
>>> the constant tables are so large as to incur lots of cache misses. scan.l
>>> is not quite as big a deal for some reason, even though it's also large.
>>>
>>> regards, tom lane
>> Yes, my results shows that pg_parse_query adds not so much overhead:
>> 206k TPS for my first variant with string literal substitution and modified query text used as hash key vs.
>> 181k. TPS for version with patching raw parse tree constructed by pg_parse_query.
> Those numbers and your statement seem to contradict each other?
Oops, my parse error:( I incorrectly read Tom's statement.
Actually, I also was afraid that price of parsing is large enough and this is why my first attempt was to avoid parsing.
But then I find out that most of the time is spent in analyze and planning (see attached profile):
pg_parse_query: 4.23%
pg_analyze_and_rewrite 8.45%
pg_plan_queries: 15.49%
>
> - Andres
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[image/svg+xml] pgbench.svg (69.9K, ../../[email protected]/2-pgbench.svg)
download | view image
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-12 00:58 Bruce Momjian <[email protected]>
parent: Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Bruce Momjian @ 2017-05-12 00:58 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On Thu, May 11, 2017 at 10:41:45PM +0300, Konstantin Knizhnik wrote:
> This is why I have provided second implementation which replace
> literals with parameters after raw parsing. Certainly it is slower
> than first approach. But still provide significant advantage in
> performance: more than two times at pgbench. Then I tried to run
> regression tests and find several situations where type analysis is
> not correctly performed in case of replacing literals with parameters.
So the issue is that per-command output from the parser, SelectStmt,
only has strings for identifers, e.g. table and column names, so you
can't be sure it is the same as the cached entry you matched. I suppose
if you cleared the cache every time someone created an object or changed
search_path, it might work.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-12 07:50 Konstantin Knizhnik <[email protected]>
parent: Bruce Momjian <[email protected]>
0 siblings, 2 replies; 51+ messages in thread
From: Konstantin Knizhnik @ 2017-05-12 07:50 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On 12.05.2017 03:58, Bruce Momjian wrote:
> On Thu, May 11, 2017 at 10:41:45PM +0300, Konstantin Knizhnik wrote:
>> This is why I have provided second implementation which replace
>> literals with parameters after raw parsing. Certainly it is slower
>> than first approach. But still provide significant advantage in
>> performance: more than two times at pgbench. Then I tried to run
>> regression tests and find several situations where type analysis is
>> not correctly performed in case of replacing literals with parameters.
> So the issue is that per-command output from the parser, SelectStmt,
> only has strings for identifers, e.g. table and column names, so you
> can't be sure it is the same as the cached entry you matched. I suppose
> if you cleared the cache every time someone created an object or changed
> search_path, it might work.
>
Definitely changing session context (search_path, date/time format, ...)
may cause incorrect behavior of cached statements.
Actually you may get the same problem with explicitly prepared
statements (certainly, in the last case, you better understand what
going on and it is your choice whether to use or not to use prepared
statement).
The fact of failure of 7 regression tests means that autoprepare can
really change behavior of existed program. This is why my suggestion is
to switch off this feature by default.
But in 99.9% real cases (my estimation plucked out of thin air:) there
will be no such problems with autoprepare. And it can significantly
improve performance of OLTP applications
which are not able to use prepared statements (because of working
through pgbouncer or any other reasons).
Can autoprepare slow down the system?
Yes, it can. It can happen if application perform larger number of
unique queries and autoprepare cache size is not limited.
In this case large (and infinitely growing) number of stored plans can
consume a lot of memory and, what is even worse, slowdown cache lookup.
This is why I by default limit number of cached statements
(autoprepare_limit parameter) by 100.
I am almost sure that there will be some other issues with autoprepare
which I have not encountered yet (because I mostly tested it on pgbench
and Postgres regression tests).
But I am also sure that benefit of doubling system performance is good
motivation to continue work in this direction.
My main concern is whether to continue to improve current approach with
local (per-backend) cache of prepared statements.
Or create shared cache (as in Oracle). It is much more difficult to
implement shared cache (the same problem with session context, different
catalog snapshots, cache invalidation,...)
but it also provides more opportunities for queries optimization and
tuning.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-12 15:23 Bruce Momjian <[email protected]>
parent: Konstantin Knizhnik <[email protected]>
1 sibling, 1 reply; 51+ messages in thread
From: Bruce Momjian @ 2017-05-12 15:23 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On Fri, May 12, 2017 at 10:50:41AM +0300, Konstantin Knizhnik wrote:
> Definitely changing session context (search_path, date/time format, ...) may
> cause incorrect behavior of cached statements.
I wonder if we should clear the cache whenever any SET command is
issued.
> Actually you may get the same problem with explicitly prepared statements
> (certainly, in the last case, you better understand what going on and it is
> your choice whether to use or not to use prepared statement).
>
> The fact of failure of 7 regression tests means that autoprepare can really
> change behavior of existed program. This is why my suggestion is to switch
> off this feature by default.
I would like to see us target something that can be enabled by default.
Even if it only improves performance by 5%, it would be better overall
than a feature that improves performance by 90% but is only used by 1%
of our users.
> But in 99.9% real cases (my estimation plucked out of thin air:) there will
> be no such problems with autoprepare. And it can significantly improve
> performance of OLTP applications
> which are not able to use prepared statements (because of working through
> pgbouncer or any other reasons).
Right, but we can't ship something that is 99.9% accurate when the
inaccuracy is indeterminate. The bottom line is that Postgres has a
very high bar, and I realize you are just prototyping at this point, but
the final product is going to have to address all the intricacies of the
issue for it to be added.
> Can autoprepare slow down the system?
> Yes, it can. It can happen if application perform larger number of unique
> queries and autoprepare cache size is not limited.
> In this case large (and infinitely growing) number of stored plans can
> consume a lot of memory and, what is even worse, slowdown cache lookup.
> This is why I by default limit number of cached statements
> (autoprepare_limit parameter) by 100.
Yes, good idea.
> I am almost sure that there will be some other issues with autoprepare which
> I have not encountered yet (because I mostly tested it on pgbench and
> Postgres regression tests).
> But I am also sure that benefit of doubling system performance is good
> motivation to continue work in this direction.
Right, you are still testing to see where the edges are.
> My main concern is whether to continue to improve current approach with
> local (per-backend) cache of prepared statements.
> Or create shared cache (as in Oracle). It is much more difficult to
> implement shared cache (the same problem with session context, different
> catalog snapshots, cache invalidation,...)
> but it also provides more opportunities for queries optimization and tuning.
I would continue in the per-backend cache direction at this point
because we don't even have that solved yet. The global cache is going
to be even more complex.
Ultimately I think we are going to want global and local caches because
the plans of the local cache are much more likely to be accurate.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-12 17:35 Konstantin Knizhnik <[email protected]>
parent: Bruce Momjian <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Konstantin Knizhnik @ 2017-05-12 17:35 UTC (permalink / raw)
To: Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On 12.05.2017 18:23, Bruce Momjian wrote:
> On Fri, May 12, 2017 at 10:50:41AM +0300, Konstantin Knizhnik wrote:
>> Definitely changing session context (search_path, date/time format, ...) may
>> cause incorrect behavior of cached statements.
> I wonder if we should clear the cache whenever any SET command is
> issued.
Well, with autoprepare cache disabled on each set variable, alter system
and any slow utility statement
only one regression test is not passed. And only because of different
error message context:
*** /home/knizhnik/postgresql.master/src/test/regress/expected/date.out
2017-04-11 18:07:56.497461208 +0300
--- /home/knizhnik/postgresql.master/src/test/regress/results/date.out
2017-05-12 20:21:19.767566302 +0300
***************
*** 1443,1452 ****
--
SELECT EXTRACT(MICROSEC FROM DATE 'infinity'); -- ERROR:
timestamp units "microsec" not recognized
ERROR: timestamp units "microsec" not recognized
- CONTEXT: SQL function "date_part" statement 1
SELECT EXTRACT(UNDEFINED FROM DATE 'infinity'); -- ERROR:
timestamp units "undefined" not supported
ERROR: timestamp units "undefined" not supported
- CONTEXT: SQL function "date_part" statement 1
-- test constructors
select make_date(2013, 7, 15);
make_date
--- 1443,1450 ----
======================================================================
>> Actually you may get the same problem with explicitly prepared statements
>> (certainly, in the last case, you better understand what going on and it is
>> your choice whether to use or not to use prepared statement).
>>
>> The fact of failure of 7 regression tests means that autoprepare can really
>> change behavior of existed program. This is why my suggestion is to switch
>> off this feature by default.
> I would like to see us target something that can be enabled by default.
> Even if it only improves performance by 5%, it would be better overall
> than a feature that improves performance by 90% but is only used by 1%
> of our users.
I have to agree with you here.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-12 17:38 Andres Freund <[email protected]>
parent: Konstantin Knizhnik <[email protected]>
1 sibling, 0 replies; 51+ messages in thread
From: Andres Freund @ 2017-05-12 17:38 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; Bruce Momjian <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers
On May 12, 2017 12:50:41 AM PDT, Konstantin Knizhnik <[email protected]> wrote:
>Definitely changing session context (search_path, date/time format,
>...)
>may cause incorrect behavior of cached statements.
>Actually you may get the same problem with explicitly prepared
>statements (certainly, in the last case, you better understand what
>going on and it is your choice whether to use or not to use prepared
>statement).
>
>The fact of failure of 7 regression tests means that autoprepare can
>really change behavior of existed program. This is why my suggestion is
>
>to switch off this feature by default.
I can't see us agreeing with this feature unless it actually reliably works, even if disabled by default.
Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-12 17:43 Bruce Momjian <[email protected]>
parent: Konstantin Knizhnik <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bruce Momjian @ 2017-05-12 17:43 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On Fri, May 12, 2017 at 08:35:26PM +0300, Konstantin Knizhnik wrote:
>
>
> On 12.05.2017 18:23, Bruce Momjian wrote:
> >On Fri, May 12, 2017 at 10:50:41AM +0300, Konstantin Knizhnik wrote:
> >>Definitely changing session context (search_path, date/time format, ...) may
> >>cause incorrect behavior of cached statements.
> >I wonder if we should clear the cache whenever any SET command is
> >issued.
>
> Well, with autoprepare cache disabled on each set variable, alter system and
> any slow utility statement
> only one regression test is not passed. And only because of different error
> message context:
Great. We have to think of how we would keep this reliable when future
changes happen. Simple is often better because it limits the odds of
breakage.
> >>Actually you may get the same problem with explicitly prepared statements
> >>(certainly, in the last case, you better understand what going on and it is
> >>your choice whether to use or not to use prepared statement).
> >>
> >>The fact of failure of 7 regression tests means that autoprepare can really
> >>change behavior of existed program. This is why my suggestion is to switch
> >>off this feature by default.
> >I would like to see us target something that can be enabled by default.
> >Even if it only improves performance by 5%, it would be better overall
> >than a feature that improves performance by 90% but is only used by 1%
> >of our users.
>
> I have to agree with you here.
Good. I know there are database systems that will try to get the most
performance possible no matter how complex it is for users to configure
or to maintain, but that isn't us.
--
Bruce Momjian <[email protected]> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-15 15:31 Robert Haas <[email protected]>
parent: Konstantin Knizhnik <[email protected]>
2 siblings, 1 reply; 51+ messages in thread
From: Robert Haas @ 2017-05-15 15:31 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On Wed, May 10, 2017 at 12:11 PM, Konstantin Knizhnik
<[email protected]> wrote:
> Robert, can you please explain why using TRY/CATCH is not safe here:
>>
>> This is definitely not a safe way of using TRY/CATCH.
This has been discussed many, many times on this mailing list before,
and I don't really want to go over it again here. We really need a
README or some documentation about this so that we don't have to keep
answering this same question again and again.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-18 08:57 Konstantin Knizhnik <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Konstantin Knizhnik @ 2017-05-18 08:57 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Pavel Stehule <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>
On 15.05.2017 18:31, Robert Haas wrote:
> On Wed, May 10, 2017 at 12:11 PM, Konstantin Knizhnik
> <[email protected]> wrote:
>> Robert, can you please explain why using TRY/CATCH is not safe here:
>>> This is definitely not a safe way of using TRY/CATCH.
> This has been discussed many, many times on this mailing list before,
> and I don't really want to go over it again here. We really need a
> README or some documentation about this so that we don't have to keep
> answering this same question again and again.
>
First of all I want to notice that new version of my patch is not using
PG_TRY/PG_CATCH.
But I still want to clarify for myself whats wrong with this constructions.
I searched both hackers mailing list archive and world-wide using google
but failed to find any references except of
sharing non-volatilie variables between try and catch blocks.
Can you please point me at the thread where this problem was discussed
or just explain in few words the source of the problem?
From my own experience I found out that PG_TRY/PG_CATCH mechanism is
not providing proper cleanup (unlike C++ exceptions).
If there are opened relations, catalog cache entries,... then throwing
error will not release them.
It will cause no problems if error is handled in PostgresMain which
aborts current transaction and releases all resources in any case.
But if I want to ignore this error and continue query execution, then
warnings about resources leaks can be reported.
Is it want you mean by unsafety of PG_TRY/PG_CATCH constructions?
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-18 16:47 Andres Freund <[email protected]>
parent: Konstantin Knizhnik <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Andres Freund @ 2017-05-18 16:47 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Robert Haas <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers
On 2017-05-18 11:57:57 +0300, Konstantin Knizhnik wrote:
> From my own experience I found out that PG_TRY/PG_CATCH mechanism is not
> providing proper cleanup (unlike C++ exceptions).
Right, simply because there's no portable way to transparently do so.
Would be possible on elf glibc platforms, but ...
> If there are opened relations, catalog cache entries,... then throwing error
> will not release them.
> It will cause no problems if error is handled in PostgresMain which aborts
> current transaction and releases all resources in any case.
> But if I want to ignore this error and continue query execution, then
> warnings about resources leaks can be reported.
> Is it want you mean by unsafety of PG_TRY/PG_CATCH constructions?
There's worse than just leaking resources. Everything touching the
database might cause persistent corruption if you don't roll back.
Consider an insert that failed with a foreign key exception, done from
some function. If you ignore that error, the row will still be visible,
but the foreign key will be violated. If you want to continue after a
PG_CATCH you have to use a subtransaction/savepoint for the PG_TRY
contents, like several PLs do.
- Andres
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-05-25 15:54 Konstantin Knizhnik <[email protected]>
parent: Konstantin Knizhnik <[email protected]>
2 siblings, 1 reply; 51+ messages in thread
From: Konstantin Knizhnik @ 2017-05-25 15:54 UTC (permalink / raw)
To: pgsql-hackers
On 10.05.2017 19:11, Konstantin Knizhnik wrote:
>
> Based on the Robert's feedback and Tom's proposal I have implemented
> two new versions of autoprepare patch.
>
> First version is just refactoring of my original implementation: I
> have extracted common code into prepare_cached_plan and
> exec_prepared_plan
> function to avoid code duplication. Also I rewrote assignment of
> values to parameters. Now types of parameters are inferred from types
> of literals, so there may be several
> prepared plans which are different only by types of parameters. Due to
> the problem with type coercion for parameters, I have to catch errors
> in parse_analyze_varparams.
>
Attached please find rebased version of the autoprepare patch based on
Tom's proposal (perform analyze for tree with constant literals and then
replace them with parameters).
Also I submitted this patch for the Autum commitfest.
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-patch] autoprepare.patch (42.8K, ../../[email protected]/2-autoprepare.patch)
download | inline diff:
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index 95c1d3e..0b0642b 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -3710,6 +3710,454 @@ raw_expression_tree_walker(Node *node,
}
/*
+ * raw_expression_tree_mutator --- transform raw parse tree.
+ *
+ * This function is implementing slightly different approach for tree update than expression_tree_mutator().
+ * Callback is given pointer to pointer to the current node and can update this field instead of returning reference to new node.
+ * It makes it possible to remember changes and easily revert them without extra traversal of the tree.
+ *
+ * This function do not need QTW_DONT_COPY_QUERY flag: it never implicitly copy tree nodes, doing in-place update.
+ *
+ * Like raw_expression_tree_walker, there is no special rule about query
+ * boundaries: we descend to everything that's possibly interesting.
+ *
+ * Currently, the node type coverage here extends only to DML statements
+ * (SELECT/INSERT/UPDATE/DELETE) and nodes that can appear in them, because
+ * this is used mainly during analysis of CTEs, and only DML statements can
+ * appear in CTEs. If some other node is visited, iteration is immediately stopped and true is returned.
+ */
+bool
+raw_expression_tree_mutator(Node *node,
+ bool (*mutator) (),
+ void *context)
+{
+ ListCell *temp;
+
+ /*
+ * The walker has already visited the current node, and so we need only
+ * recurse into any sub-nodes it has.
+ */
+ if (node == NULL)
+ return false;
+
+ /* Guard against stack overflow due to overly complex expressions */
+ check_stack_depth();
+
+ switch (nodeTag(node))
+ {
+ case T_SetToDefault:
+ case T_CurrentOfExpr:
+ case T_Integer:
+ case T_Float:
+ case T_String:
+ case T_BitString:
+ case T_Null:
+ case T_ParamRef:
+ case T_A_Const:
+ case T_A_Star:
+ /* primitive node types with no subnodes */
+ break;
+ case T_Alias:
+ /* we assume the colnames list isn't interesting */
+ break;
+ case T_RangeVar:
+ return mutator(&((RangeVar *) node)->alias, context);
+ case T_GroupingFunc:
+ return mutator(&((GroupingFunc *) node)->args, context);
+ case T_SubLink:
+ {
+ SubLink *sublink = (SubLink *) node;
+
+ if (mutator(&sublink->testexpr, context))
+ return true;
+ /* we assume the operName is not interesting */
+ if (mutator(&sublink->subselect, context))
+ return true;
+ }
+ break;
+ case T_CaseExpr:
+ {
+ CaseExpr *caseexpr = (CaseExpr *) node;
+
+ if (mutator(&caseexpr->arg, context))
+ return true;
+ /* we assume mutator(& doesn't care about CaseWhens, either */
+ foreach(temp, caseexpr->args)
+ {
+ CaseWhen *when = (CaseWhen *) lfirst(temp);
+
+ Assert(IsA(when, CaseWhen));
+ if (mutator(&when->expr, context))
+ return true;
+ if (mutator(&when->result, context))
+ return true;
+ }
+ if (mutator(&caseexpr->defresult, context))
+ return true;
+ }
+ break;
+ case T_RowExpr:
+ /* Assume colnames isn't interesting */
+ return mutator(&((RowExpr *) node)->args, context);
+ case T_CoalesceExpr:
+ return mutator(&((CoalesceExpr *) node)->args, context);
+ case T_MinMaxExpr:
+ return mutator(&((MinMaxExpr *) node)->args, context);
+ case T_XmlExpr:
+ {
+ XmlExpr *xexpr = (XmlExpr *) node;
+
+ if (mutator(&xexpr->named_args, context))
+ return true;
+ /* we assume mutator(& doesn't care about arg_names */
+ if (mutator(&xexpr->args, context))
+ return true;
+ }
+ break;
+ case T_NullTest:
+ return mutator(&((NullTest *) node)->arg, context);
+ case T_BooleanTest:
+ return mutator(&((BooleanTest *) node)->arg, context);
+ case T_JoinExpr:
+ {
+ JoinExpr *join = (JoinExpr *) node;
+
+ if (mutator(&join->larg, context))
+ return true;
+ if (mutator(&join->rarg, context))
+ return true;
+ if (mutator(&join->quals, context))
+ return true;
+ if (mutator(&join->alias, context))
+ return true;
+ /* using list is deemed uninteresting */
+ }
+ break;
+ case T_IntoClause:
+ {
+ IntoClause *into = (IntoClause *) node;
+
+ if (mutator(&into->rel, context))
+ return true;
+ /* colNames, options are deemed uninteresting */
+ /* viewQuery should be null in raw parsetree, but check it */
+ if (mutator(&into->viewQuery, context))
+ return true;
+ }
+ break;
+ case T_List:
+ foreach(temp, (List *) node)
+ {
+ if (mutator(&lfirst(temp), context))
+ return true;
+ }
+ break;
+ case T_InsertStmt:
+ {
+ InsertStmt *stmt = (InsertStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->cols, context))
+ return true;
+ if (mutator(&stmt->selectStmt, context))
+ return true;
+ if (mutator(&stmt->onConflictClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_DeleteStmt:
+ {
+ DeleteStmt *stmt = (DeleteStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->usingClause, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_UpdateStmt:
+ {
+ UpdateStmt *stmt = (UpdateStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->fromClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_SelectStmt:
+ {
+ SelectStmt *stmt = (SelectStmt *) node;
+
+ if (mutator(&stmt->distinctClause, context))
+ return true;
+ if (mutator(&stmt->intoClause, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->fromClause, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->groupClause, context))
+ return true;
+ if (mutator(&stmt->havingClause, context))
+ return true;
+ if (mutator(&stmt->windowClause, context))
+ return true;
+ if (mutator(&stmt->valuesLists, context))
+ return true;
+ if (mutator(&stmt->sortClause, context))
+ return true;
+ if (mutator(&stmt->limitOffset, context))
+ return true;
+ if (mutator(&stmt->limitCount, context))
+ return true;
+ if (mutator(&stmt->lockingClause, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ if (mutator(&stmt->larg, context))
+ return true;
+ if (mutator(&stmt->rarg, context))
+ return true;
+ }
+ break;
+ case T_A_Expr:
+ {
+ A_Expr *expr = (A_Expr *) node;
+
+ if (mutator(&expr->lexpr, context))
+ return true;
+ if (mutator(&expr->rexpr, context))
+ return true;
+ /* operator name is deemed uninteresting */
+ }
+ break;
+ case T_BoolExpr:
+ {
+ BoolExpr *expr = (BoolExpr *) node;
+
+ if (mutator(&expr->args, context))
+ return true;
+ }
+ break;
+ case T_ColumnRef:
+ /* we assume the fields contain nothing interesting */
+ break;
+ case T_FuncCall:
+ {
+ FuncCall *fcall = (FuncCall *) node;
+
+ if (mutator(&fcall->args, context))
+ return true;
+ if (mutator(&fcall->agg_order, context))
+ return true;
+ if (mutator(&fcall->agg_filter, context))
+ return true;
+ if (mutator(&fcall->over, context))
+ return true;
+ /* function name is deemed uninteresting */
+ }
+ break;
+ case T_NamedArgExpr:
+ return mutator(&((NamedArgExpr *) node)->arg, context);
+ case T_A_Indices:
+ {
+ A_Indices *indices = (A_Indices *) node;
+
+ if (mutator(&indices->lidx, context))
+ return true;
+ if (mutator(&indices->uidx, context))
+ return true;
+ }
+ break;
+ case T_A_Indirection:
+ {
+ A_Indirection *indir = (A_Indirection *) node;
+
+ if (mutator(&indir->arg, context))
+ return true;
+ if (mutator(&indir->indirection, context))
+ return true;
+ }
+ break;
+ case T_A_ArrayExpr:
+ return mutator(&((A_ArrayExpr *) node)->elements, context);
+ case T_ResTarget:
+ {
+ ResTarget *rt = (ResTarget *) node;
+
+ if (mutator(&rt->indirection, context))
+ return true;
+ if (mutator(&rt->val, context))
+ return true;
+ }
+ break;
+ case T_MultiAssignRef:
+ return mutator(&((MultiAssignRef *) node)->source, context);
+ case T_TypeCast:
+ {
+ TypeCast *tc = (TypeCast *) node;
+
+ if (mutator(&tc->arg, context))
+ return true;
+ if (mutator(&tc->typeName, context))
+ return true;
+ }
+ break;
+ case T_CollateClause:
+ return mutator(&((CollateClause *) node)->arg, context);
+ case T_SortBy:
+ return mutator(&((SortBy *) node)->node, context);
+ case T_WindowDef:
+ {
+ WindowDef *wd = (WindowDef *) node;
+
+ if (mutator(&wd->partitionClause, context))
+ return true;
+ if (mutator(&wd->orderClause, context))
+ return true;
+ if (mutator(&wd->startOffset, context))
+ return true;
+ if (mutator(&wd->endOffset, context))
+ return true;
+ }
+ break;
+ case T_RangeSubselect:
+ {
+ RangeSubselect *rs = (RangeSubselect *) node;
+
+ if (mutator(&rs->subquery, context))
+ return true;
+ if (mutator(&rs->alias, context))
+ return true;
+ }
+ break;
+ case T_RangeFunction:
+ {
+ RangeFunction *rf = (RangeFunction *) node;
+
+ if (mutator(&rf->functions, context))
+ return true;
+ if (mutator(&rf->alias, context))
+ return true;
+ if (mutator(&rf->coldeflist, context))
+ return true;
+ }
+ break;
+ case T_RangeTableSample:
+ {
+ RangeTableSample *rts = (RangeTableSample *) node;
+
+ if (mutator(&rts->relation, context))
+ return true;
+ /* method name is deemed uninteresting */
+ if (mutator(&rts->args, context))
+ return true;
+ if (mutator(&rts->repeatable, context))
+ return true;
+ }
+ break;
+ case T_TypeName:
+ {
+ TypeName *tn = (TypeName *) node;
+
+ if (mutator(&tn->typmods, context))
+ return true;
+ if (mutator(&tn->arrayBounds, context))
+ return true;
+ /* type name itself is deemed uninteresting */
+ }
+ break;
+ case T_ColumnDef:
+ {
+ ColumnDef *coldef = (ColumnDef *) node;
+
+ if (mutator(&coldef->typeName, context))
+ return true;
+ if (mutator(&coldef->raw_default, context))
+ return true;
+ if (mutator(&coldef->collClause, context))
+ return true;
+ /* for now, constraints are ignored */
+ }
+ break;
+ case T_IndexElem:
+ {
+ IndexElem *indelem = (IndexElem *) node;
+
+ if (mutator(&indelem->expr, context))
+ return true;
+ /* collation and opclass names are deemed uninteresting */
+ }
+ break;
+ case T_GroupingSet:
+ return mutator(&((GroupingSet *) node)->content, context);
+ case T_LockingClause:
+ return mutator(&((LockingClause *) node)->lockedRels, context);
+ case T_XmlSerialize:
+ {
+ XmlSerialize *xs = (XmlSerialize *) node;
+
+ if (mutator(&xs->expr, context))
+ return true;
+ if (mutator(&xs->typeName, context))
+ return true;
+ }
+ break;
+ case T_WithClause:
+ return mutator(&((WithClause *) node)->ctes, context);
+ case T_InferClause:
+ {
+ InferClause *stmt = (InferClause *) node;
+
+ if (mutator(&stmt->indexElems, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ }
+ break;
+ case T_OnConflictClause:
+ {
+ OnConflictClause *stmt = (OnConflictClause *) node;
+
+ if (mutator(&stmt->infer, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ }
+ break;
+ case T_CommonTableExpr:
+ return mutator(&((CommonTableExpr *) node)->ctequery, context);
+ default:
+ return true;
+ }
+ return false;
+}
+
+/*
* planstate_tree_walker --- walk plan state trees
*
* The walker has already visited the current node, and so we need only
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 75c2d9a..abab8c3 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -42,11 +42,13 @@
#include "catalog/pg_type.h"
#include "commands/async.h"
#include "commands/prepare.h"
+#include "commands/defrem.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "nodes/print.h"
+#include "nodes/nodeFuncs.h"
#include "optimizer/planner.h"
#include "pgstat.h"
#include "pg_trace.h"
@@ -73,6 +75,7 @@
#include "utils/snapmgr.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
+#include "utils/int8.h"
#include "mb/pg_wchar.h"
@@ -188,7 +191,8 @@ static bool IsTransactionStmtList(List *pstmts);
static void drop_unnamed_stmt(void);
static void SigHupHandler(SIGNAL_ARGS);
static void log_disconnections(int code, Datum arg);
-
+static bool exec_cached_query(const char* query, List *parsetree_list);
+static void exec_prepared_plan(Portal portal, const char *portal_name, long max_rows, CommandDest dest);
/* ----------------------------------------------------------------
* routines to obtain user input
@@ -962,6 +966,14 @@ exec_simple_query(const char *query_string)
isTopLevel = (list_length(parsetree_list) == 1);
/*
+ * Try to find cached plan
+ */
+ if (isTopLevel && autoprepare_threshold != 0 && exec_cached_query(query_string, parsetree_list))
+ {
+ return;
+ }
+
+ /*
* Run through the raw parsetree(s) and process each one.
*/
foreach(parsetree_item, parsetree_list)
@@ -1845,9 +1857,28 @@ exec_bind_message(StringInfo input_message)
static void
exec_execute_message(const char *portal_name, long max_rows)
{
- CommandDest dest;
+ Portal portal = GetPortalByName(portal_name);
+ CommandDest dest = whereToSendOutput;
+
+ /* Adjust destination to tell printtup.c what to do */
+ if (dest == DestRemote)
+ dest = DestRemoteExecute;
+
+ if (!PortalIsValid(portal))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_CURSOR),
+ errmsg("portal \"%s\" does not exist", portal_name)));
+
+ exec_prepared_plan(portal, portal_name, max_rows, dest);
+}
+
+/*
+ * Execute prepared plan.
+ */
+static void
+exec_prepared_plan(Portal portal, const char *portal_name, long max_rows, CommandDest dest)
+{
DestReceiver *receiver;
- Portal portal;
bool completed;
char completionTag[COMPLETION_TAG_BUFSIZE];
const char *sourceText;
@@ -1859,17 +1890,6 @@ exec_execute_message(const char *portal_name, long max_rows)
bool was_logged = false;
char msec_str[32];
- /* Adjust destination to tell printtup.c what to do */
- dest = whereToSendOutput;
- if (dest == DestRemote)
- dest = DestRemoteExecute;
-
- portal = GetPortalByName(portal_name);
- if (!PortalIsValid(portal))
- ereport(ERROR,
- (errcode(ERRCODE_UNDEFINED_CURSOR),
- errmsg("portal \"%s\" does not exist", portal_name)));
-
/*
* If the original query was a null string, just return
* EmptyQueryResponse.
@@ -1932,7 +1952,7 @@ exec_execute_message(const char *portal_name, long max_rows)
* context, because that may get deleted if portal contains VACUUM).
*/
receiver = CreateDestReceiver(dest);
- if (dest == DestRemoteExecute)
+ if (dest == DestRemoteExecute || dest == DestRemote)
SetRemoteDestReceiverParams(receiver, portal);
/*
@@ -4499,3 +4519,782 @@ log_disconnections(int code, Datum arg)
port->user_name, port->database_name, port->remote_host,
port->remote_port[0] ? " port=" : "", port->remote_port)));
}
+
+/*
+ * Autoprepare implementation.
+ * Autoprepare consists of raw parse tree mutator, hash table of cached plans and exec_cached_query function
+ * which combines exec_parse_message + exec_bind_message + exec_execute_message
+ */
+
+/*
+ * Mapping between parameters and replaced literals
+ */
+typedef struct ParamBinding
+{
+ A_Const* literal; /* Original literal */
+ ParamRef* paramref;/* Constructed parameter reference */
+ Param* param; /* Constructed parameter */
+ Node** ref; /* Pointer to pointer to literal node (used to revert raw parse tree update) */
+ Oid raw_type;/* Parameter raw type */
+ Oid type; /* Parameter type after analysis */
+ struct ParamBinding* next; /* L1-list of query parameter bindings */
+} ParamBinding;
+
+/*
+ * Plan cache entry
+ */
+typedef struct
+{
+ Node* parse_tree; /* tree is used as hash key */
+ dlist_node lru; /* double linked list to implement LRU */
+ int64 exec_count; /* counter of execution of this query */
+ CachedPlanSource* plan;
+ uint32 hash; /* hash calculated for this parsed tree */
+ Oid* param_types;/* types of parameters */
+ int n_params; /* number of parameters extracted for this query */
+ int16 format; /* portal output format */
+ bool disable_autoprepare; /* disable preparing of this query */
+} plan_cache_entry;
+
+static uint32 plan_cache_hash_fn(const void *key, Size keysize)
+{
+ return ((plan_cache_entry*)key)->hash;
+}
+
+static int plan_cache_match_fn(const void *key1, const void *key2, Size keysize)
+{
+ plan_cache_entry* e1 = (plan_cache_entry*)key1;
+ plan_cache_entry* e2 = (plan_cache_entry*)key2;
+
+ return equal(e1->parse_tree, e2->parse_tree)
+ && memcmp(e1->param_types, e2->param_types, sizeof(Oid)*e1->n_params) == 0 ? 0 : 1;
+}
+
+static void* plan_cache_keycopy_fn(void *dest, const void *src, Size keysize)
+{
+ plan_cache_entry* dst_entry = (plan_cache_entry*)dest;
+ plan_cache_entry* src_entry = (plan_cache_entry*)src;
+ dst_entry->parse_tree = copyObject(src_entry->parse_tree);
+ dst_entry->param_types = palloc(src_entry->n_params*sizeof(Oid));
+ dst_entry->n_params = src_entry->n_params;
+ memcpy(dst_entry->param_types, src_entry->param_types, src_entry->n_params*sizeof(Oid));
+ dst_entry->hash = src_entry->hash;
+ return dest;
+}
+
+#define PLAN_CACHE_SIZE 113
+
+/*
+ * Plan cache access statistic
+ */
+size_t autoprepare_hits;
+size_t autoprepare_misses;
+size_t autoprepare_cached_plans;
+
+/*
+ * Context for raw_expression_tree_mutator
+ */
+typedef struct {
+ int n_params; /* Number of extracted parameters */
+ uint32 hash; /* We calculate hash for parse tree during plan traversal */
+ ParamBinding** param_list_tail; /* pointer to last element "next" field address, used to construct L1 list of parameters */
+} GeneralizerCtx;
+
+
+static HTAB* plan_cache_hash; /* hash table for plan cache */
+static dlist_head plan_cache_lru; /* LRU L2-list for cached queries */
+static MemoryContext plan_cache_context; /* memory context used for plan cache */
+
+/*
+ * Check if expression is constant (used to eliminate substitution of literals with parameters in such expressions
+ */
+static bool is_constant_expression(Node* node)
+{
+ return node != NULL
+ && (IsA(node, A_Const)
+ || (IsA(node, A_Expr)
+ && is_constant_expression(((A_Expr*)node)->lexpr)
+ && is_constant_expression(((A_Expr*)node)->rexpr)));
+}
+
+/*
+ * Infer type of literal expression. Null literals should not be replaced with parameters.
+ */
+static Oid get_literal_type(Value* val)
+{
+ int64 val64;
+ switch (val->type)
+ {
+ case T_Integer:
+ return INT4OID;
+ case T_Float:
+ /* could be an oversize integer as well as a float ... */
+ if (scanint8(strVal(val), true, &val64))
+ {
+ /*
+ * It might actually fit in int32. Probably only INT_MIN can
+ * occur, but we'll code the test generally just to be sure.
+ */
+ int32 val32 = (int32) val64;
+ return (val64 == (int64)val32) ? INT4OID : INT8OID;
+ }
+ else
+ {
+ return NUMERICOID;
+ }
+ case T_BitString:
+ return BITOID;
+ case T_String:
+ return UNKNOWNOID;
+ default:
+ Assert(false);
+ return InvalidOid;
+ }
+}
+
+static Datum get_param_value(Oid type, Value* val)
+{
+ if (val->type == T_Integer && type == INT4OID)
+ {
+ /*
+ * Integer constant
+ */
+ return Int32GetDatum((int32)val->val.ival);
+ }
+ else
+ {
+ /*
+ * Convert from string literal
+ */
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(type, &typinput, &typioparam);
+ return OidInputFunctionCall(typinput, val->val.str, typioparam, -1);
+ }
+}
+
+
+/*
+ * Callback for raw_expression_tree_mutator performing substitution of literals with parameters
+ */
+static bool
+raw_parse_tree_generalizer(Node** ref, void *context)
+{
+ Node* node = *ref;
+ GeneralizerCtx* ctx = (GeneralizerCtx*)context;
+ if (node == NULL)
+ {
+ return false;
+ }
+ /*
+ * Calculate hash for parse tree. We consider only node tags here, precise comparison of trees is done using equal() function.
+ * Here we calculate hash for original (unpatched) tree, without ParamRef nodes.
+ * It is non principle, because hash calculation doesn't take in account types and values of Const nodes. So the same generalized queries
+ * will have the same hash value. There are about 1000 different nodes tags, this is why we rotate hash on 10 bits.
+ */
+ ctx->hash = (ctx->hash << 10) ^ (ctx->hash >> 22) ^ nodeTag(node);
+
+ switch (nodeTag(node))
+ {
+ case T_A_Expr:
+ {
+ /*
+ * Do not perform substitution of literals in constant expression (which is likely to be the same for all queries and optimized by compiler)
+ */
+ if (!is_constant_expression(node))
+ {
+ A_Expr *expr = (A_Expr *) node;
+ if (raw_parse_tree_generalizer((Node**)&expr->lexpr, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&expr->rexpr, context))
+ return true;
+ }
+ break;
+ }
+ case T_A_Const:
+ {
+ /*
+ * Do substitution of literals with parameters here
+ */
+ A_Const* literal = (A_Const*)node;
+ if (literal->val.type != T_Null)
+ {
+ /*
+ * Do not substitute null literals with parameters
+ */
+ ParamBinding* cp = palloc0(sizeof(ParamBinding));
+ ParamRef* param = makeNode(ParamRef);
+ param->number = ++ctx->n_params;
+ param->location = literal->location;
+ cp->ref = ref;
+ cp->paramref = param;
+ cp->literal = literal;
+ cp->raw_type = get_literal_type(&literal->val);
+ *ctx->param_list_tail = cp;
+ ctx->param_list_tail = &cp->next;
+ *ref = (Node*)param;
+ }
+ break;
+ }
+ case T_SelectStmt:
+ {
+ /*
+ * Substitute literals only in target list, WHERE, VALUES and WITH clause,
+ * skipping target and from lists, which is unlikely contains some parameterized values
+ */
+ SelectStmt *stmt = (SelectStmt *) node;
+ if (stmt->intoClause)
+ return true; /* Utility statement can not be prepared */
+ if (raw_parse_tree_generalizer((Node**)&stmt->targetList, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->whereClause, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->valuesLists, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->withClause, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->larg, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->rarg, context))
+ return true;
+ break;
+ }
+ case T_TypeName:
+ case T_SortGroupClause:
+ case T_SortBy:
+ case T_A_ArrayExpr:
+ case T_TypeCast:
+ /*
+ * Literals in this clauses should not be replaced with parameters
+ */
+ break;
+ default:
+ /*
+ * Default traversal. raw_expression_tree_mutator returns true for all not recognized nodes, for example right now
+ * all transaction control statements are not covered by raw_expression_tree_mutator and so will not auto prepared.
+ * My experiments show that effect of non-preparing start/commit transaction statements is positive.
+ */
+ return raw_expression_tree_mutator(node, raw_parse_tree_generalizer, context);
+ }
+ return false;
+}
+
+static Node*
+parse_tree_generalizer(Node *node, void *context)
+{
+ ParamBinding* binding;
+ ParamBinding* binding_list = (ParamBinding*)context;
+ if (node == NULL)
+ {
+ return NULL;
+ }
+ if (IsA(node, Query))
+ {
+ return (Node*)query_tree_mutator((Query*)node,
+ parse_tree_generalizer,
+ context,
+ QTW_DONT_COPY_QUERY);
+ }
+ if (IsA(node, Const))
+ {
+ Const* c = (Const*)node;
+ int paramno = 1;
+ for (binding = binding_list; binding != NULL && binding->literal->location != c->location; binding = binding->next, paramno++);
+ if (binding != NULL)
+ {
+ if (binding->param != NULL)
+ {
+ /* Parameter can be used only once */
+ binding->type = UNKNOWNOID;
+ //return (Node*)binding->param;
+ }
+ else
+ {
+ Param* param = makeNode(Param);
+ param->paramkind = PARAM_EXTERN;
+ param->paramid = paramno;
+ param->paramtype = c->consttype;
+ param->paramtypmod = c->consttypmod;
+ param->paramcollid = c->constcollid;
+ param->location = c->location;
+ binding->type = c->consttype;
+ binding->param = param;
+ return (Node*)param;
+ }
+ }
+ return node;
+ }
+ return expression_tree_mutator(node, parse_tree_generalizer, context);
+}
+
+/*
+ * Restore original parse tree, replacing all ParamRef back with Const nodes.
+ * Such undo operation seems to be more efficient than copying the whole parse tree by raw_expression_tree_mutator
+ */
+static void undo_query_plan_changes(ParamBinding* cp)
+{
+ while (cp != NULL) {
+ *cp->ref = (Node*)cp->literal;
+ cp = cp->next;
+ }
+}
+
+/*
+ * Callback for raw_expression_tree_walker dropping parse tree
+ */
+static bool drop_tree_node(Node* node, void* context)
+{
+ if (node) {
+ raw_expression_tree_walker(node, drop_tree_node, NULL);
+ pfree(node);
+ }
+ return false;
+}
+
+/*
+ * Location of converted literal in query.
+ * Used for precise error reporting (line number)
+ */
+static int param_location;
+
+/*
+ * Error callback adding information about error location
+ */
+static void
+prepare_error_callback(void *arg)
+{
+ CachedPlanSource *psrc = (CachedPlanSource*)arg;
+ /* And pass it to the ereport mechanism */
+ if (geterrcode() != ERRCODE_QUERY_CANCELED) {
+ int pos = pg_mbstrlen_with_len(psrc->query_string, param_location) + 1;
+ (void)errposition(pos);
+ }
+}
+/*
+ * Try to generalize query, find cached plan for it and execute
+ */
+static bool exec_cached_query(const char *query_string, List *parsetree_list)
+{
+ int n_params;
+ plan_cache_entry *entry;
+ bool found;
+ MemoryContext old_context;
+ CachedPlanSource *psrc;
+ ParamListInfo params;
+ int paramno;
+ CachedPlan *cplan;
+ Portal portal;
+ bool snapshot_set = false;
+ GeneralizerCtx ctx;
+ ParamBinding* binding;
+ ParamBinding* binding_list;
+ plan_cache_entry pattern;
+ Oid* param_types;
+ RawStmt *raw_parse_tree;
+
+ raw_parse_tree = castNode(RawStmt, linitial(parsetree_list));
+
+ /*
+ * Substitute literals with parameters and calculate hash for raw parse tree
+ */
+ ctx.param_list_tail = &binding_list;
+ ctx.n_params = 0;
+ ctx.hash = 0;
+ if (raw_parse_tree_generalizer(&raw_parse_tree->stmt, &ctx))
+ {
+ *ctx.param_list_tail = NULL;
+ undo_query_plan_changes(binding_list);
+ autoprepare_misses += 1;
+ return false;
+ }
+ *ctx.param_list_tail = NULL;
+ n_params = ctx.n_params;
+
+ /*
+ * Extract array of parameters types: it is needed for cached plan lookup
+ */
+ param_types = (Oid*)palloc(sizeof(Oid)*n_params);
+ for (paramno = 0, binding = binding_list; paramno < n_params; paramno++, binding = binding->next)
+ {
+ param_types[paramno] = binding->raw_type;
+ }
+
+ /*
+ * Construct plan cache context if not constructed yet.
+ */
+ if (plan_cache_context == NULL)
+ {
+ plan_cache_context = AllocSetContextCreate(TopMemoryContext,
+ "plan cache context",
+ ALLOCSET_DEFAULT_SIZES);
+ }
+ /* Manipulations with hash table are performed in plan_cache_context memory context */
+ old_context = MemoryContextSwitchTo(plan_cache_context);
+
+ /*
+ * Initialize hash table if not initialized yet
+ */
+ if (plan_cache_hash == NULL)
+ {
+ static HASHCTL info;
+ info.keysize = sizeof(plan_cache_entry);
+ info.entrysize = sizeof(plan_cache_entry);
+ info.hash = plan_cache_hash_fn;
+ info.match = plan_cache_match_fn;
+ info.keycopy = plan_cache_keycopy_fn;
+ plan_cache_hash = hash_create("plan_cache", autoprepare_limit != 0 ? autoprepare_limit : PLAN_CACHE_SIZE,
+ &info, HASH_ELEM | HASH_FUNCTION | HASH_COMPARE | HASH_KEYCOPY);
+ dlist_init(&plan_cache_lru);
+ }
+
+ /*
+ * Lookup generalized query
+ */
+ pattern.parse_tree = raw_parse_tree->stmt;
+ pattern.hash = ctx.hash;
+ pattern.n_params = n_params;
+ pattern.param_types = param_types;
+ entry = (plan_cache_entry*)hash_search(plan_cache_hash, &pattern, HASH_ENTER, &found);
+ if (!found)
+ {
+ /* Check number of cached queries */
+ if (++autoprepare_cached_plans > autoprepare_limit && autoprepare_limit != 0)
+ {
+ /* Drop least recently accessed query */
+ plan_cache_entry* victim = dlist_container(plan_cache_entry, lru, plan_cache_lru.head.prev);
+ Node* dropped_tree = victim->parse_tree;
+ dlist_delete(&victim->lru);
+ if (victim->plan)
+ {
+ DropCachedPlan(victim->plan);
+ }
+ pfree(victim->param_types);
+ hash_search(plan_cache_hash, victim, HASH_REMOVE, NULL);
+ raw_expression_tree_walker(dropped_tree, drop_tree_node, NULL);
+ autoprepare_cached_plans -= 1;
+ }
+ entry->exec_count = 0;
+ entry->plan = NULL;
+ entry->disable_autoprepare = false;
+ }
+ else
+ {
+ dlist_delete(&entry->lru); /* accessed entry will be moved to the head of LRU list */
+ if (entry->plan != NULL && !entry->plan->is_valid)
+ {
+ /* Drop invalidated plan: it will be reconstructed later */
+ DropCachedPlan(entry->plan);
+ entry->plan = NULL;
+ }
+ }
+ dlist_insert_after(&plan_cache_lru.head, &entry->lru); /* prepend entry to the head of LRU list */
+ MemoryContextSwitchTo(old_context); /* Done with plan_cache_context memory context */
+
+
+ /*
+ * Prepare query only when it is executed more than autoprepare_threshold times
+ */
+ if (entry->disable_autoprepare || entry->exec_count++ < autoprepare_threshold)
+ {
+ undo_query_plan_changes(binding_list);
+ autoprepare_misses += 1;
+ return false;
+ }
+
+ if (entry->plan == NULL)
+ {
+ bool snapshot_set = false;
+ const char *commandTag;
+ List *querytree_list;
+
+ /*
+ * Switch to appropriate context for preparing plan.
+ */
+ old_context = MemoryContextSwitchTo(MessageContext);
+
+ /*
+ * Get the command name for use in status display (it also becomes the
+ * default completion tag, down inside PortalRun). Set ps_status and
+ * do any special start-of-SQL-command processing needed by the
+ * destination.
+ */
+ commandTag = CreateCommandTag(raw_parse_tree->stmt);
+
+ /*
+ * If we are in an aborted transaction, reject all commands except
+ * COMMIT/ABORT. It is important that this test occur before we try
+ * to do parse analysis, rewrite, or planning, since all those phases
+ * try to do database accesses, which may fail in abort state. (It
+ * might be safe to allow some additional utility commands in this
+ * state, but not many...)
+ */
+ if (IsAbortedTransactionBlockState() &&
+ !IsTransactionExitStmt(raw_parse_tree->stmt))
+ ereport(ERROR,
+ (errcode(ERRCODE_IN_FAILED_SQL_TRANSACTION),
+ errmsg("current transaction is aborted, "
+ "commands ignored until end of transaction block"),
+ errdetail_abort()));
+
+ /*
+ * Create the CachedPlanSource before we do parse analysis, since it
+ * needs to see the unmodified raw parse tree.
+ */
+ psrc = CreateCachedPlan(raw_parse_tree, query_string, commandTag);
+
+ /*
+ * Revert raw plan to use literals
+ */
+ undo_query_plan_changes(binding_list);
+
+ /*
+ * Set up a snapshot if parse analysis/planning will need one.
+ */
+ if (analyze_requires_snapshot(raw_parse_tree))
+ {
+ PushActiveSnapshot(GetTransactionSnapshot());
+ snapshot_set = true;
+ }
+
+ querytree_list = pg_analyze_and_rewrite(raw_parse_tree, query_string,
+ NULL, 0);
+ /*
+ * Replace Const with Param nodes
+ */
+ (void)query_tree_mutator((Query*)linitial(querytree_list),
+ parse_tree_generalizer,
+ binding_list,
+ QTW_DONT_COPY_QUERY);
+
+ /* Done with the snapshot used for parsing/planning */
+ if (snapshot_set)
+ PopActiveSnapshot();
+
+ param_types = (Oid*)palloc(sizeof(Oid)*n_params);
+ psrc->param_types = param_types;
+ for (paramno = 0, binding = binding_list; paramno < n_params; paramno++, binding = binding->next)
+ {
+ if (binding->param == NULL || binding->type == UNKNOWNOID)
+ {
+ /* Failed to resolve parameter type */
+ entry->disable_autoprepare = true;
+ autoprepare_misses += 1;
+ MemoryContextSwitchTo(old_context);
+ return false;
+ }
+ param_types[paramno] = binding->type;
+ }
+
+ /* Finish filling in the CachedPlanSource */
+ CompleteCachedPlan(psrc,
+ querytree_list,
+ NULL,
+ param_types,
+ n_params,
+ NULL,
+ NULL,
+ CURSOR_OPT_PARALLEL_OK, /* allow parallel mode */
+ true); /* fixed result */
+
+ /* If we got a cancel signal during analysis, quit */
+ CHECK_FOR_INTERRUPTS();
+
+ SaveCachedPlan(psrc);
+
+ /*
+ * We do NOT close the open transaction command here; that only happens
+ * when the client sends Sync. Instead, do CommandCounterIncrement just
+ * in case something happened during parse/plan.
+ */
+ CommandCounterIncrement();
+
+ MemoryContextSwitchTo(old_context); /* Done with MessageContext memory context */
+
+ entry->plan = psrc;
+
+ /*
+ * Determine output format
+ */
+ entry->format = 0; /* TEXT is default */
+ if (IsA(raw_parse_tree->stmt, FetchStmt))
+ {
+ FetchStmt *stmt = (FetchStmt *)raw_parse_tree->stmt;
+
+ if (!stmt->ismove)
+ {
+ Portal fportal = GetPortalByName(stmt->portalname);
+
+ if (PortalIsValid(fportal) &&
+ (fportal->cursorOptions & CURSOR_OPT_BINARY))
+ entry->format = 1; /* BINARY */
+ }
+ }
+ }
+ else
+ {
+ /* Plan found */
+ psrc = entry->plan;
+ Assert(n_params == entry->n_params);
+ }
+
+ /*
+ * If we are in aborted transaction state, the only portals we can
+ * actually run are those containing COMMIT or ROLLBACK commands. We
+ * disallow binding anything else to avoid problems with infrastructure
+ * that expects to run inside a valid transaction. We also disallow
+ * binding any parameters, since we can't risk calling user-defined I/O
+ * functions.
+ */
+ if (IsAbortedTransactionBlockState() &&
+ (!IsTransactionExitStmt(psrc->raw_parse_tree->stmt) ||
+ n_params != 0))
+ ereport(ERROR,
+ (errcode(ERRCODE_IN_FAILED_SQL_TRANSACTION),
+ errmsg("current transaction is aborted, "
+ "commands ignored until end of transaction block"),
+ errdetail_abort()));
+
+ /*
+ * Create unnamed portal to run the query or queries in. If there
+ * already is one, silently drop it.
+ */
+ portal = CreatePortal("", true, true);
+ /* Don't display the portal in pg_cursors */
+ portal->visible = false;
+
+ /*
+ * Prepare to copy stuff into the portal's memory context. We do all this
+ * copying first, because it could possibly fail (out-of-memory) and we
+ * don't want a failure to occur between GetCachedPlan and
+ * PortalDefineQuery; that would result in leaking our plancache refcount.
+ */
+ old_context = MemoryContextSwitchTo(PortalGetHeapMemory(portal));
+
+ /* Copy the plan's query string into the portal */
+ query_string = pstrdup(psrc->query_string);
+
+ /*
+ * Set a snapshot if we have parameters to fetch (since the input
+ * functions might need it) or the query isn't a utility command (and
+ * hence could require redoing parse analysis and planning). We keep the
+ * snapshot active till we're done, so that plancache.c doesn't have to
+ * take new ones.
+ */
+ if (n_params > 0 ||
+ (psrc->raw_parse_tree &&
+ analyze_requires_snapshot(psrc->raw_parse_tree)))
+ {
+ PushActiveSnapshot(GetTransactionSnapshot());
+ snapshot_set = true;
+ }
+
+ /*
+ * Fetch parameters, if any, and store in the portal's memory context.
+ */
+ if (n_params > 0)
+ {
+ ErrorContextCallback errcallback;
+
+ params = (ParamListInfo) palloc0(offsetof(ParamListInfoData, params) +
+ n_params * sizeof(ParamExternData));
+ params->numParams = n_params;
+
+ /*
+ * Register error callback to precisely report error in case of conversion error while storig parameter value.
+ */
+ errcallback.callback = prepare_error_callback;
+ errcallback.arg = (void *) psrc;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ for (paramno = 0, binding = binding_list;
+ paramno < n_params;
+ paramno++, binding = binding->next)
+ {
+ Oid ptype = psrc->param_types[paramno];
+
+ param_location = binding->literal->location;
+
+ params->params[paramno].isnull = false;
+ params->params[paramno].value = get_param_value(ptype, &binding->literal->val);
+ /*
+ * We mark the params as CONST. This ensures that any custom plan
+ * makes full use of the parameter values.
+ */
+ params->params[paramno].pflags = PARAM_FLAG_CONST;
+ params->params[paramno].ptype = ptype;
+ }
+ error_context_stack = errcallback.previous;
+ }
+ else
+ {
+ params = NULL;
+ }
+
+ /* Done storing stuff in portal's context */
+ MemoryContextSwitchTo(old_context);
+
+ /*
+ * Obtain a plan from the CachedPlanSource. Any cruft from (re)planning
+ * will be generated in MessageContext. The plan refcount will be
+ * assigned to the Portal, so it will be released at portal destruction.
+ */
+ cplan = GetCachedPlan(psrc, params, false);
+
+ /*
+ * Now we can define the portal.
+ *
+ * DO NOT put any code that could possibly throw an error between the
+ * above GetCachedPlan call and here.
+ */
+ PortalDefineQuery(portal,
+ NULL,
+ query_string,
+ psrc->commandTag,
+ cplan->stmt_list,
+ cplan);
+
+ /* Done with the snapshot used for parameter I/O and parsing/planning */
+ if (snapshot_set)
+ {
+ PopActiveSnapshot();
+ }
+
+ /*
+ * And we're ready to start portal execution.
+ */
+ PortalStart(portal, params, 0, InvalidSnapshot);
+
+ /*
+ * Apply the result format requests to the portal.
+ */
+ PortalSetResultFormat(portal, 1, &entry->format);
+
+ /*
+ * Finally execute prepared statement
+ */
+ exec_prepared_plan(portal, "", FETCH_ALL, whereToSendOutput);
+
+ /*
+ * Close down transaction statement, if one is open.
+ */
+ finish_xact_command();
+
+ autoprepare_hits += 1;
+
+ return true;
+}
+
+
+void ResetAutoprepareCache(void)
+{
+ if (plan_cache_hash != NULL)
+ {
+ hash_destroy(plan_cache_hash);
+ MemoryContextReset(plan_cache_context);
+ dlist_init(&plan_cache_lru);
+ autoprepare_cached_plans = 0;
+ plan_cache_hash = 0;
+ }
+}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 1e941fb..8735650 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -684,10 +684,12 @@ standard_ProcessUtility(PlannedStmt *pstmt,
case T_AlterSystemStmt:
PreventTransactionChain(isTopLevel, "ALTER SYSTEM");
AlterSystemSetConfigFile((AlterSystemStmt *) parsetree);
+ ResetAutoprepareCache();
break;
case T_VariableSetStmt:
ExecSetVariableStmt((VariableSetStmt *) parsetree, isTopLevel);
+ ResetAutoprepareCache();
break;
case T_VariableShowStmt:
@@ -960,6 +962,8 @@ ProcessUtilitySlow(ParseState *pstate,
/* All event trigger calls are done only when isCompleteQuery is true */
needCleanup = isCompleteQuery && EventTriggerBeginCompleteQuery();
+ ResetAutoprepareCache();
+
/* PG_TRY block is to ensure we call EventTriggerEndCompleteQuery */
PG_TRY();
{
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 8191216..a3e6b9d 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -111,6 +111,7 @@
#include "utils/relmapper.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+#include "tcop/pquery.h"
/*
@@ -644,6 +645,7 @@ InvalidateSystemCaches(void)
InvalidateCatalogSnapshot();
ResetCatalogCaches();
+ ResetAutoprepareCache();
RelationCacheInvalidate(); /* gets smgr and relmap too */
for (i = 0; i < syscache_callback_count; i++)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 92e1d63..ee673b4 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -472,6 +472,10 @@ int tcp_keepalives_idle;
int tcp_keepalives_interval;
int tcp_keepalives_count;
+
+int autoprepare_threshold;
+int autoprepare_limit;
+
/*
* SSL renegotiation was been removed in PostgreSQL 9.5, but we tolerate it
* being set to zero (meaning never renegotiate) for backward compatibility.
@@ -1958,6 +1962,28 @@ static struct config_int ConfigureNamesInt[] =
check_max_stack_depth, assign_max_stack_depth, NULL
},
+ /*
+ * Threshold for implicit preparing of frequently executed queries
+ */
+ {
+ {"autoprepare_threshold", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Threshold for autopreparing query."),
+ gettext_noop("0 value disables autoprepare.")
+ },
+ &autoprepare_threshold,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+ {
+ {"autoprepare_limit", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Maximal number of autoprepared queries."),
+ gettext_noop("0 means unlimited number of autoprepared queries. Too large number of prepared queries can cause backend memory overflow and slowdown execution speed (because of increased lookup time)")
+ },
+ &autoprepare_limit,
+ 113, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"temp_file_limit", PGC_SUSET, RESOURCES_DISK,
gettext_noop("Limits the total size of all temporary files used by each process."),
diff --git a/src/include/nodes/nodeFuncs.h b/src/include/nodes/nodeFuncs.h
index b6c9b48..1388822 100644
--- a/src/include/nodes/nodeFuncs.h
+++ b/src/include/nodes/nodeFuncs.h
@@ -73,8 +73,11 @@ extern Node *query_or_expression_tree_mutator(Node *node, Node *(*mutator) (),
extern bool raw_expression_tree_walker(Node *node, bool (*walker) (),
void *context);
+extern bool raw_expression_tree_mutator(Node *node, bool (*mutator) (),
+ void *context);
struct PlanState;
extern bool planstate_tree_walker(struct PlanState *planstate, bool (*walker) (),
void *context);
+
#endif /* NODEFUNCS_H */
diff --git a/src/include/tcop/pquery.h b/src/include/tcop/pquery.h
index 12ff458..50dfad6 100644
--- a/src/include/tcop/pquery.h
+++ b/src/include/tcop/pquery.h
@@ -42,4 +42,6 @@ extern uint64 PortalRunFetch(Portal portal,
long count,
DestReceiver *dest);
+extern void ResetAutoprepareCache(void);
+
#endif /* PQUERY_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 87d0741..1215ac8 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -251,6 +251,9 @@ extern int client_min_messages;
extern int log_min_duration_statement;
extern int log_temp_files;
+extern int autoprepare_threshold;
+extern int autoprepare_limit;
+
extern int temp_file_limit;
extern int num_temp_buffers;
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-09-09 03:35 Thomas Munro <[email protected]>
parent: Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Thomas Munro @ 2017-09-09 03:35 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: pgsql-hackers
On Fri, May 26, 2017 at 3:54 AM, Konstantin Knizhnik
<[email protected]> wrote:
> Attached please find rebased version of the autoprepare patch based on Tom's
> proposal (perform analyze for tree with constant literals and then replace
> them with parameters).
> Also I submitted this patch for the Autum commitfest.
The patch didn't survive the Summer bitrotfest. Could you please rebase it?
--
Thomas Munro
http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-09-11 09:24 Konstantin Knizhnik <[email protected]>
parent: Thomas Munro <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Konstantin Knizhnik @ 2017-09-11 09:24 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: pgsql-hackers
On 09.09.2017 06:35, Thomas Munro wrote:
> On Fri, May 26, 2017 at 3:54 AM, Konstantin Knizhnik
> <[email protected]> wrote:
>> Attached please find rebased version of the autoprepare patch based on Tom's
>> proposal (perform analyze for tree with constant literals and then replace
>> them with parameters).
>> Also I submitted this patch for the Autum commitfest.
> The patch didn't survive the Summer bitrotfest. Could you please rebase it?
>
Attached please find rebased version of the patch.
There are the updated performance results (pgbench -s 100 -c 1):
protocol (-M)
read-write
read-only (-S)
simple
3327
19325
extended
2256
16908
prepared
6145
39326
simple+autoprepare
4950
34841
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-patch] autoprepare-3.patch (42.7K, ../../[email protected]/3-autoprepare-3.patch)
download | inline diff:
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index e3eb0c5..17f3dfd 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -3688,6 +3688,454 @@ raw_expression_tree_walker(Node *node,
}
/*
+ * raw_expression_tree_mutator --- transform raw parse tree.
+ *
+ * This function is implementing slightly different approach for tree update than expression_tree_mutator().
+ * Callback is given pointer to pointer to the current node and can update this field instead of returning reference to new node.
+ * It makes it possible to remember changes and easily revert them without extra traversal of the tree.
+ *
+ * This function do not need QTW_DONT_COPY_QUERY flag: it never implicitly copy tree nodes, doing in-place update.
+ *
+ * Like raw_expression_tree_walker, there is no special rule about query
+ * boundaries: we descend to everything that's possibly interesting.
+ *
+ * Currently, the node type coverage here extends only to DML statements
+ * (SELECT/INSERT/UPDATE/DELETE) and nodes that can appear in them, because
+ * this is used mainly during analysis of CTEs, and only DML statements can
+ * appear in CTEs. If some other node is visited, iteration is immediately stopped and true is returned.
+ */
+bool
+raw_expression_tree_mutator(Node *node,
+ bool (*mutator) (),
+ void *context)
+{
+ ListCell *temp;
+
+ /*
+ * The walker has already visited the current node, and so we need only
+ * recurse into any sub-nodes it has.
+ */
+ if (node == NULL)
+ return false;
+
+ /* Guard against stack overflow due to overly complex expressions */
+ check_stack_depth();
+
+ switch (nodeTag(node))
+ {
+ case T_SetToDefault:
+ case T_CurrentOfExpr:
+ case T_Integer:
+ case T_Float:
+ case T_String:
+ case T_BitString:
+ case T_Null:
+ case T_ParamRef:
+ case T_A_Const:
+ case T_A_Star:
+ /* primitive node types with no subnodes */
+ break;
+ case T_Alias:
+ /* we assume the colnames list isn't interesting */
+ break;
+ case T_RangeVar:
+ return mutator(&((RangeVar *) node)->alias, context);
+ case T_GroupingFunc:
+ return mutator(&((GroupingFunc *) node)->args, context);
+ case T_SubLink:
+ {
+ SubLink *sublink = (SubLink *) node;
+
+ if (mutator(&sublink->testexpr, context))
+ return true;
+ /* we assume the operName is not interesting */
+ if (mutator(&sublink->subselect, context))
+ return true;
+ }
+ break;
+ case T_CaseExpr:
+ {
+ CaseExpr *caseexpr = (CaseExpr *) node;
+
+ if (mutator(&caseexpr->arg, context))
+ return true;
+ /* we assume mutator(& doesn't care about CaseWhens, either */
+ foreach(temp, caseexpr->args)
+ {
+ CaseWhen *when = (CaseWhen *) lfirst(temp);
+
+ Assert(IsA(when, CaseWhen));
+ if (mutator(&when->expr, context))
+ return true;
+ if (mutator(&when->result, context))
+ return true;
+ }
+ if (mutator(&caseexpr->defresult, context))
+ return true;
+ }
+ break;
+ case T_RowExpr:
+ /* Assume colnames isn't interesting */
+ return mutator(&((RowExpr *) node)->args, context);
+ case T_CoalesceExpr:
+ return mutator(&((CoalesceExpr *) node)->args, context);
+ case T_MinMaxExpr:
+ return mutator(&((MinMaxExpr *) node)->args, context);
+ case T_XmlExpr:
+ {
+ XmlExpr *xexpr = (XmlExpr *) node;
+
+ if (mutator(&xexpr->named_args, context))
+ return true;
+ /* we assume mutator(& doesn't care about arg_names */
+ if (mutator(&xexpr->args, context))
+ return true;
+ }
+ break;
+ case T_NullTest:
+ return mutator(&((NullTest *) node)->arg, context);
+ case T_BooleanTest:
+ return mutator(&((BooleanTest *) node)->arg, context);
+ case T_JoinExpr:
+ {
+ JoinExpr *join = (JoinExpr *) node;
+
+ if (mutator(&join->larg, context))
+ return true;
+ if (mutator(&join->rarg, context))
+ return true;
+ if (mutator(&join->quals, context))
+ return true;
+ if (mutator(&join->alias, context))
+ return true;
+ /* using list is deemed uninteresting */
+ }
+ break;
+ case T_IntoClause:
+ {
+ IntoClause *into = (IntoClause *) node;
+
+ if (mutator(&into->rel, context))
+ return true;
+ /* colNames, options are deemed uninteresting */
+ /* viewQuery should be null in raw parsetree, but check it */
+ if (mutator(&into->viewQuery, context))
+ return true;
+ }
+ break;
+ case T_List:
+ foreach(temp, (List *) node)
+ {
+ if (mutator(&lfirst(temp), context))
+ return true;
+ }
+ break;
+ case T_InsertStmt:
+ {
+ InsertStmt *stmt = (InsertStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->cols, context))
+ return true;
+ if (mutator(&stmt->selectStmt, context))
+ return true;
+ if (mutator(&stmt->onConflictClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_DeleteStmt:
+ {
+ DeleteStmt *stmt = (DeleteStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->usingClause, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_UpdateStmt:
+ {
+ UpdateStmt *stmt = (UpdateStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->fromClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_SelectStmt:
+ {
+ SelectStmt *stmt = (SelectStmt *) node;
+
+ if (mutator(&stmt->distinctClause, context))
+ return true;
+ if (mutator(&stmt->intoClause, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->fromClause, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->groupClause, context))
+ return true;
+ if (mutator(&stmt->havingClause, context))
+ return true;
+ if (mutator(&stmt->windowClause, context))
+ return true;
+ if (mutator(&stmt->valuesLists, context))
+ return true;
+ if (mutator(&stmt->sortClause, context))
+ return true;
+ if (mutator(&stmt->limitOffset, context))
+ return true;
+ if (mutator(&stmt->limitCount, context))
+ return true;
+ if (mutator(&stmt->lockingClause, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ if (mutator(&stmt->larg, context))
+ return true;
+ if (mutator(&stmt->rarg, context))
+ return true;
+ }
+ break;
+ case T_A_Expr:
+ {
+ A_Expr *expr = (A_Expr *) node;
+
+ if (mutator(&expr->lexpr, context))
+ return true;
+ if (mutator(&expr->rexpr, context))
+ return true;
+ /* operator name is deemed uninteresting */
+ }
+ break;
+ case T_BoolExpr:
+ {
+ BoolExpr *expr = (BoolExpr *) node;
+
+ if (mutator(&expr->args, context))
+ return true;
+ }
+ break;
+ case T_ColumnRef:
+ /* we assume the fields contain nothing interesting */
+ break;
+ case T_FuncCall:
+ {
+ FuncCall *fcall = (FuncCall *) node;
+
+ if (mutator(&fcall->args, context))
+ return true;
+ if (mutator(&fcall->agg_order, context))
+ return true;
+ if (mutator(&fcall->agg_filter, context))
+ return true;
+ if (mutator(&fcall->over, context))
+ return true;
+ /* function name is deemed uninteresting */
+ }
+ break;
+ case T_NamedArgExpr:
+ return mutator(&((NamedArgExpr *) node)->arg, context);
+ case T_A_Indices:
+ {
+ A_Indices *indices = (A_Indices *) node;
+
+ if (mutator(&indices->lidx, context))
+ return true;
+ if (mutator(&indices->uidx, context))
+ return true;
+ }
+ break;
+ case T_A_Indirection:
+ {
+ A_Indirection *indir = (A_Indirection *) node;
+
+ if (mutator(&indir->arg, context))
+ return true;
+ if (mutator(&indir->indirection, context))
+ return true;
+ }
+ break;
+ case T_A_ArrayExpr:
+ return mutator(&((A_ArrayExpr *) node)->elements, context);
+ case T_ResTarget:
+ {
+ ResTarget *rt = (ResTarget *) node;
+
+ if (mutator(&rt->indirection, context))
+ return true;
+ if (mutator(&rt->val, context))
+ return true;
+ }
+ break;
+ case T_MultiAssignRef:
+ return mutator(&((MultiAssignRef *) node)->source, context);
+ case T_TypeCast:
+ {
+ TypeCast *tc = (TypeCast *) node;
+
+ if (mutator(&tc->arg, context))
+ return true;
+ if (mutator(&tc->typeName, context))
+ return true;
+ }
+ break;
+ case T_CollateClause:
+ return mutator(&((CollateClause *) node)->arg, context);
+ case T_SortBy:
+ return mutator(&((SortBy *) node)->node, context);
+ case T_WindowDef:
+ {
+ WindowDef *wd = (WindowDef *) node;
+
+ if (mutator(&wd->partitionClause, context))
+ return true;
+ if (mutator(&wd->orderClause, context))
+ return true;
+ if (mutator(&wd->startOffset, context))
+ return true;
+ if (mutator(&wd->endOffset, context))
+ return true;
+ }
+ break;
+ case T_RangeSubselect:
+ {
+ RangeSubselect *rs = (RangeSubselect *) node;
+
+ if (mutator(&rs->subquery, context))
+ return true;
+ if (mutator(&rs->alias, context))
+ return true;
+ }
+ break;
+ case T_RangeFunction:
+ {
+ RangeFunction *rf = (RangeFunction *) node;
+
+ if (mutator(&rf->functions, context))
+ return true;
+ if (mutator(&rf->alias, context))
+ return true;
+ if (mutator(&rf->coldeflist, context))
+ return true;
+ }
+ break;
+ case T_RangeTableSample:
+ {
+ RangeTableSample *rts = (RangeTableSample *) node;
+
+ if (mutator(&rts->relation, context))
+ return true;
+ /* method name is deemed uninteresting */
+ if (mutator(&rts->args, context))
+ return true;
+ if (mutator(&rts->repeatable, context))
+ return true;
+ }
+ break;
+ case T_TypeName:
+ {
+ TypeName *tn = (TypeName *) node;
+
+ if (mutator(&tn->typmods, context))
+ return true;
+ if (mutator(&tn->arrayBounds, context))
+ return true;
+ /* type name itself is deemed uninteresting */
+ }
+ break;
+ case T_ColumnDef:
+ {
+ ColumnDef *coldef = (ColumnDef *) node;
+
+ if (mutator(&coldef->typeName, context))
+ return true;
+ if (mutator(&coldef->raw_default, context))
+ return true;
+ if (mutator(&coldef->collClause, context))
+ return true;
+ /* for now, constraints are ignored */
+ }
+ break;
+ case T_IndexElem:
+ {
+ IndexElem *indelem = (IndexElem *) node;
+
+ if (mutator(&indelem->expr, context))
+ return true;
+ /* collation and opclass names are deemed uninteresting */
+ }
+ break;
+ case T_GroupingSet:
+ return mutator(&((GroupingSet *) node)->content, context);
+ case T_LockingClause:
+ return mutator(&((LockingClause *) node)->lockedRels, context);
+ case T_XmlSerialize:
+ {
+ XmlSerialize *xs = (XmlSerialize *) node;
+
+ if (mutator(&xs->expr, context))
+ return true;
+ if (mutator(&xs->typeName, context))
+ return true;
+ }
+ break;
+ case T_WithClause:
+ return mutator(&((WithClause *) node)->ctes, context);
+ case T_InferClause:
+ {
+ InferClause *stmt = (InferClause *) node;
+
+ if (mutator(&stmt->indexElems, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ }
+ break;
+ case T_OnConflictClause:
+ {
+ OnConflictClause *stmt = (OnConflictClause *) node;
+
+ if (mutator(&stmt->infer, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ }
+ break;
+ case T_CommonTableExpr:
+ return mutator(&((CommonTableExpr *) node)->ctequery, context);
+ default:
+ return true;
+ }
+ return false;
+}
+
+/*
* planstate_tree_walker --- walk plan state trees
*
* The walker has already visited the current node, and so we need only
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index c10d891..d41c0e5 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -42,11 +42,13 @@
#include "catalog/pg_type.h"
#include "commands/async.h"
#include "commands/prepare.h"
+#include "commands/defrem.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "nodes/print.h"
+#include "nodes/nodeFuncs.h"
#include "optimizer/planner.h"
#include "pgstat.h"
#include "pg_trace.h"
@@ -75,6 +77,7 @@
#include "utils/snapmgr.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
+#include "utils/int8.h"
#include "mb/pg_wchar.h"
@@ -182,7 +185,8 @@ static bool IsTransactionExitStmtList(List *pstmts);
static bool IsTransactionStmtList(List *pstmts);
static void drop_unnamed_stmt(void);
static void log_disconnections(int code, Datum arg);
-
+static bool exec_cached_query(const char* query, List *parsetree_list);
+static void exec_prepared_plan(Portal portal, const char *portal_name, long max_rows, CommandDest dest);
/* ----------------------------------------------------------------
* routines to obtain user input
@@ -956,6 +960,14 @@ exec_simple_query(const char *query_string)
use_implicit_block = (list_length(parsetree_list) > 1);
/*
+ * Try to find cached plan
+ */
+ if (autoprepare_threshold != 0 && exec_cached_query(query_string, parsetree_list))
+ {
+ return;
+ }
+
+ /*
* Run through the raw parsetree(s) and process each one.
*/
foreach(parsetree_item, parsetree_list)
@@ -1852,9 +1864,28 @@ exec_bind_message(StringInfo input_message)
static void
exec_execute_message(const char *portal_name, long max_rows)
{
- CommandDest dest;
+ Portal portal = GetPortalByName(portal_name);
+ CommandDest dest = whereToSendOutput;
+
+ /* Adjust destination to tell printtup.c what to do */
+ if (dest == DestRemote)
+ dest = DestRemoteExecute;
+
+ if (!PortalIsValid(portal))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_CURSOR),
+ errmsg("portal \"%s\" does not exist", portal_name)));
+
+ exec_prepared_plan(portal, portal_name, max_rows, dest);
+}
+
+/*
+ * Execute prepared plan.
+ */
+static void
+exec_prepared_plan(Portal portal, const char *portal_name, long max_rows, CommandDest dest)
+{
DestReceiver *receiver;
- Portal portal;
bool completed;
char completionTag[COMPLETION_TAG_BUFSIZE];
const char *sourceText;
@@ -1866,17 +1897,6 @@ exec_execute_message(const char *portal_name, long max_rows)
bool was_logged = false;
char msec_str[32];
- /* Adjust destination to tell printtup.c what to do */
- dest = whereToSendOutput;
- if (dest == DestRemote)
- dest = DestRemoteExecute;
-
- portal = GetPortalByName(portal_name);
- if (!PortalIsValid(portal))
- ereport(ERROR,
- (errcode(ERRCODE_UNDEFINED_CURSOR),
- errmsg("portal \"%s\" does not exist", portal_name)));
-
/*
* If the original query was a null string, just return
* EmptyQueryResponse.
@@ -1939,7 +1959,7 @@ exec_execute_message(const char *portal_name, long max_rows)
* context, because that may get deleted if portal contains VACUUM).
*/
receiver = CreateDestReceiver(dest);
- if (dest == DestRemoteExecute)
+ if (dest == DestRemoteExecute || dest == DestRemote)
SetRemoteDestReceiverParams(receiver, portal);
/*
@@ -4534,3 +4554,782 @@ log_disconnections(int code, Datum arg)
port->user_name, port->database_name, port->remote_host,
port->remote_port[0] ? " port=" : "", port->remote_port)));
}
+
+/*
+ * Autoprepare implementation.
+ * Autoprepare consists of raw parse tree mutator, hash table of cached plans and exec_cached_query function
+ * which combines exec_parse_message + exec_bind_message + exec_execute_message
+ */
+
+/*
+ * Mapping between parameters and replaced literals
+ */
+typedef struct ParamBinding
+{
+ A_Const* literal; /* Original literal */
+ ParamRef* paramref;/* Constructed parameter reference */
+ Param* param; /* Constructed parameter */
+ Node** ref; /* Pointer to pointer to literal node (used to revert raw parse tree update) */
+ Oid raw_type;/* Parameter raw type */
+ Oid type; /* Parameter type after analysis */
+ struct ParamBinding* next; /* L1-list of query parameter bindings */
+} ParamBinding;
+
+/*
+ * Plan cache entry
+ */
+typedef struct
+{
+ Node* parse_tree; /* tree is used as hash key */
+ dlist_node lru; /* double linked list to implement LRU */
+ int64 exec_count; /* counter of execution of this query */
+ CachedPlanSource* plan;
+ uint32 hash; /* hash calculated for this parsed tree */
+ Oid* param_types;/* types of parameters */
+ int n_params; /* number of parameters extracted for this query */
+ int16 format; /* portal output format */
+ bool disable_autoprepare; /* disable preparing of this query */
+} plan_cache_entry;
+
+static uint32 plan_cache_hash_fn(const void *key, Size keysize)
+{
+ return ((plan_cache_entry*)key)->hash;
+}
+
+static int plan_cache_match_fn(const void *key1, const void *key2, Size keysize)
+{
+ plan_cache_entry* e1 = (plan_cache_entry*)key1;
+ plan_cache_entry* e2 = (plan_cache_entry*)key2;
+
+ return equal(e1->parse_tree, e2->parse_tree)
+ && memcmp(e1->param_types, e2->param_types, sizeof(Oid)*e1->n_params) == 0 ? 0 : 1;
+}
+
+static void* plan_cache_keycopy_fn(void *dest, const void *src, Size keysize)
+{
+ plan_cache_entry* dst_entry = (plan_cache_entry*)dest;
+ plan_cache_entry* src_entry = (plan_cache_entry*)src;
+ dst_entry->parse_tree = copyObject(src_entry->parse_tree);
+ dst_entry->param_types = palloc(src_entry->n_params*sizeof(Oid));
+ dst_entry->n_params = src_entry->n_params;
+ memcpy(dst_entry->param_types, src_entry->param_types, src_entry->n_params*sizeof(Oid));
+ dst_entry->hash = src_entry->hash;
+ return dest;
+}
+
+#define PLAN_CACHE_SIZE 113
+
+/*
+ * Plan cache access statistic
+ */
+size_t autoprepare_hits;
+size_t autoprepare_misses;
+size_t autoprepare_cached_plans;
+
+/*
+ * Context for raw_expression_tree_mutator
+ */
+typedef struct {
+ int n_params; /* Number of extracted parameters */
+ uint32 hash; /* We calculate hash for parse tree during plan traversal */
+ ParamBinding** param_list_tail; /* pointer to last element "next" field address, used to construct L1 list of parameters */
+} GeneralizerCtx;
+
+
+static HTAB* plan_cache_hash; /* hash table for plan cache */
+static dlist_head plan_cache_lru; /* LRU L2-list for cached queries */
+static MemoryContext plan_cache_context; /* memory context used for plan cache */
+
+/*
+ * Check if expression is constant (used to eliminate substitution of literals with parameters in such expressions
+ */
+static bool is_constant_expression(Node* node)
+{
+ return node != NULL
+ && (IsA(node, A_Const)
+ || (IsA(node, A_Expr)
+ && is_constant_expression(((A_Expr*)node)->lexpr)
+ && is_constant_expression(((A_Expr*)node)->rexpr)));
+}
+
+/*
+ * Infer type of literal expression. Null literals should not be replaced with parameters.
+ */
+static Oid get_literal_type(Value* val)
+{
+ int64 val64;
+ switch (val->type)
+ {
+ case T_Integer:
+ return INT4OID;
+ case T_Float:
+ /* could be an oversize integer as well as a float ... */
+ if (scanint8(strVal(val), true, &val64))
+ {
+ /*
+ * It might actually fit in int32. Probably only INT_MIN can
+ * occur, but we'll code the test generally just to be sure.
+ */
+ int32 val32 = (int32) val64;
+ return (val64 == (int64)val32) ? INT4OID : INT8OID;
+ }
+ else
+ {
+ return NUMERICOID;
+ }
+ case T_BitString:
+ return BITOID;
+ case T_String:
+ return UNKNOWNOID;
+ default:
+ Assert(false);
+ return InvalidOid;
+ }
+}
+
+static Datum get_param_value(Oid type, Value* val)
+{
+ if (val->type == T_Integer && type == INT4OID)
+ {
+ /*
+ * Integer constant
+ */
+ return Int32GetDatum((int32)val->val.ival);
+ }
+ else
+ {
+ /*
+ * Convert from string literal
+ */
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(type, &typinput, &typioparam);
+ return OidInputFunctionCall(typinput, val->val.str, typioparam, -1);
+ }
+}
+
+
+/*
+ * Callback for raw_expression_tree_mutator performing substitution of literals with parameters
+ */
+static bool
+raw_parse_tree_generalizer(Node** ref, void *context)
+{
+ Node* node = *ref;
+ GeneralizerCtx* ctx = (GeneralizerCtx*)context;
+ if (node == NULL)
+ {
+ return false;
+ }
+ /*
+ * Calculate hash for parse tree. We consider only node tags here, precise comparison of trees is done using equal() function.
+ * Here we calculate hash for original (unpatched) tree, without ParamRef nodes.
+ * It is non principle, because hash calculation doesn't take in account types and values of Const nodes. So the same generalized queries
+ * will have the same hash value. There are about 1000 different nodes tags, this is why we rotate hash on 10 bits.
+ */
+ ctx->hash = (ctx->hash << 10) ^ (ctx->hash >> 22) ^ nodeTag(node);
+
+ switch (nodeTag(node))
+ {
+ case T_A_Expr:
+ {
+ /*
+ * Do not perform substitution of literals in constant expression (which is likely to be the same for all queries and optimized by compiler)
+ */
+ if (!is_constant_expression(node))
+ {
+ A_Expr *expr = (A_Expr *) node;
+ if (raw_parse_tree_generalizer((Node**)&expr->lexpr, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&expr->rexpr, context))
+ return true;
+ }
+ break;
+ }
+ case T_A_Const:
+ {
+ /*
+ * Do substitution of literals with parameters here
+ */
+ A_Const* literal = (A_Const*)node;
+ if (literal->val.type != T_Null)
+ {
+ /*
+ * Do not substitute null literals with parameters
+ */
+ ParamBinding* cp = palloc0(sizeof(ParamBinding));
+ ParamRef* param = makeNode(ParamRef);
+ param->number = ++ctx->n_params;
+ param->location = literal->location;
+ cp->ref = ref;
+ cp->paramref = param;
+ cp->literal = literal;
+ cp->raw_type = get_literal_type(&literal->val);
+ *ctx->param_list_tail = cp;
+ ctx->param_list_tail = &cp->next;
+ *ref = (Node*)param;
+ }
+ break;
+ }
+ case T_SelectStmt:
+ {
+ /*
+ * Substitute literals only in target list, WHERE, VALUES and WITH clause,
+ * skipping target and from lists, which is unlikely contains some parameterized values
+ */
+ SelectStmt *stmt = (SelectStmt *) node;
+ if (stmt->intoClause)
+ return true; /* Utility statement can not be prepared */
+ if (raw_parse_tree_generalizer((Node**)&stmt->targetList, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->whereClause, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->valuesLists, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->withClause, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->larg, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->rarg, context))
+ return true;
+ break;
+ }
+ case T_TypeName:
+ case T_SortGroupClause:
+ case T_SortBy:
+ case T_A_ArrayExpr:
+ case T_TypeCast:
+ /*
+ * Literals in this clauses should not be replaced with parameters
+ */
+ break;
+ default:
+ /*
+ * Default traversal. raw_expression_tree_mutator returns true for all not recognized nodes, for example right now
+ * all transaction control statements are not covered by raw_expression_tree_mutator and so will not auto prepared.
+ * My experiments show that effect of non-preparing start/commit transaction statements is positive.
+ */
+ return raw_expression_tree_mutator(node, raw_parse_tree_generalizer, context);
+ }
+ return false;
+}
+
+static Node*
+parse_tree_generalizer(Node *node, void *context)
+{
+ ParamBinding* binding;
+ ParamBinding* binding_list = (ParamBinding*)context;
+ if (node == NULL)
+ {
+ return NULL;
+ }
+ if (IsA(node, Query))
+ {
+ return (Node*)query_tree_mutator((Query*)node,
+ parse_tree_generalizer,
+ context,
+ QTW_DONT_COPY_QUERY);
+ }
+ if (IsA(node, Const))
+ {
+ Const* c = (Const*)node;
+ int paramno = 1;
+ for (binding = binding_list; binding != NULL && binding->literal->location != c->location; binding = binding->next, paramno++);
+ if (binding != NULL)
+ {
+ if (binding->param != NULL)
+ {
+ /* Parameter can be used only once */
+ binding->type = UNKNOWNOID;
+ //return (Node*)binding->param;
+ }
+ else
+ {
+ Param* param = makeNode(Param);
+ param->paramkind = PARAM_EXTERN;
+ param->paramid = paramno;
+ param->paramtype = c->consttype;
+ param->paramtypmod = c->consttypmod;
+ param->paramcollid = c->constcollid;
+ param->location = c->location;
+ binding->type = c->consttype;
+ binding->param = param;
+ return (Node*)param;
+ }
+ }
+ return node;
+ }
+ return expression_tree_mutator(node, parse_tree_generalizer, context);
+}
+
+/*
+ * Restore original parse tree, replacing all ParamRef back with Const nodes.
+ * Such undo operation seems to be more efficient than copying the whole parse tree by raw_expression_tree_mutator
+ */
+static void undo_query_plan_changes(ParamBinding* cp)
+{
+ while (cp != NULL) {
+ *cp->ref = (Node*)cp->literal;
+ cp = cp->next;
+ }
+}
+
+/*
+ * Callback for raw_expression_tree_walker dropping parse tree
+ */
+static bool drop_tree_node(Node* node, void* context)
+{
+ if (node) {
+ raw_expression_tree_walker(node, drop_tree_node, NULL);
+ pfree(node);
+ }
+ return false;
+}
+
+/*
+ * Location of converted literal in query.
+ * Used for precise error reporting (line number)
+ */
+static int param_location;
+
+/*
+ * Error callback adding information about error location
+ */
+static void
+prepare_error_callback(void *arg)
+{
+ CachedPlanSource *psrc = (CachedPlanSource*)arg;
+ /* And pass it to the ereport mechanism */
+ if (geterrcode() != ERRCODE_QUERY_CANCELED) {
+ int pos = pg_mbstrlen_with_len(psrc->query_string, param_location) + 1;
+ (void)errposition(pos);
+ }
+}
+/*
+ * Try to generalize query, find cached plan for it and execute
+ */
+static bool exec_cached_query(const char *query_string, List *parsetree_list)
+{
+ int n_params;
+ plan_cache_entry *entry;
+ bool found;
+ MemoryContext old_context;
+ CachedPlanSource *psrc;
+ ParamListInfo params;
+ int paramno;
+ CachedPlan *cplan;
+ Portal portal;
+ bool snapshot_set = false;
+ GeneralizerCtx ctx;
+ ParamBinding* binding;
+ ParamBinding* binding_list;
+ plan_cache_entry pattern;
+ Oid* param_types;
+ RawStmt *raw_parse_tree;
+
+ raw_parse_tree = castNode(RawStmt, linitial(parsetree_list));
+
+ /*
+ * Substitute literals with parameters and calculate hash for raw parse tree
+ */
+ ctx.param_list_tail = &binding_list;
+ ctx.n_params = 0;
+ ctx.hash = 0;
+ if (raw_parse_tree_generalizer(&raw_parse_tree->stmt, &ctx))
+ {
+ *ctx.param_list_tail = NULL;
+ undo_query_plan_changes(binding_list);
+ autoprepare_misses += 1;
+ return false;
+ }
+ *ctx.param_list_tail = NULL;
+ n_params = ctx.n_params;
+
+ /*
+ * Extract array of parameters types: it is needed for cached plan lookup
+ */
+ param_types = (Oid*)palloc(sizeof(Oid)*n_params);
+ for (paramno = 0, binding = binding_list; paramno < n_params; paramno++, binding = binding->next)
+ {
+ param_types[paramno] = binding->raw_type;
+ }
+
+ /*
+ * Construct plan cache context if not constructed yet.
+ */
+ if (plan_cache_context == NULL)
+ {
+ plan_cache_context = AllocSetContextCreate(TopMemoryContext,
+ "plan cache context",
+ ALLOCSET_DEFAULT_SIZES);
+ }
+ /* Manipulations with hash table are performed in plan_cache_context memory context */
+ old_context = MemoryContextSwitchTo(plan_cache_context);
+
+ /*
+ * Initialize hash table if not initialized yet
+ */
+ if (plan_cache_hash == NULL)
+ {
+ static HASHCTL info;
+ info.keysize = sizeof(plan_cache_entry);
+ info.entrysize = sizeof(plan_cache_entry);
+ info.hash = plan_cache_hash_fn;
+ info.match = plan_cache_match_fn;
+ info.keycopy = plan_cache_keycopy_fn;
+ plan_cache_hash = hash_create("plan_cache", autoprepare_limit != 0 ? autoprepare_limit : PLAN_CACHE_SIZE,
+ &info, HASH_ELEM | HASH_FUNCTION | HASH_COMPARE | HASH_KEYCOPY);
+ dlist_init(&plan_cache_lru);
+ }
+
+ /*
+ * Lookup generalized query
+ */
+ pattern.parse_tree = raw_parse_tree->stmt;
+ pattern.hash = ctx.hash;
+ pattern.n_params = n_params;
+ pattern.param_types = param_types;
+ entry = (plan_cache_entry*)hash_search(plan_cache_hash, &pattern, HASH_ENTER, &found);
+ if (!found)
+ {
+ /* Check number of cached queries */
+ if (++autoprepare_cached_plans > autoprepare_limit && autoprepare_limit != 0)
+ {
+ /* Drop least recently accessed query */
+ plan_cache_entry* victim = dlist_container(plan_cache_entry, lru, plan_cache_lru.head.prev);
+ Node* dropped_tree = victim->parse_tree;
+ dlist_delete(&victim->lru);
+ if (victim->plan)
+ {
+ DropCachedPlan(victim->plan);
+ }
+ pfree(victim->param_types);
+ hash_search(plan_cache_hash, victim, HASH_REMOVE, NULL);
+ raw_expression_tree_walker(dropped_tree, drop_tree_node, NULL);
+ autoprepare_cached_plans -= 1;
+ }
+ entry->exec_count = 0;
+ entry->plan = NULL;
+ entry->disable_autoprepare = false;
+ }
+ else
+ {
+ dlist_delete(&entry->lru); /* accessed entry will be moved to the head of LRU list */
+ if (entry->plan != NULL && !entry->plan->is_valid)
+ {
+ /* Drop invalidated plan: it will be reconstructed later */
+ DropCachedPlan(entry->plan);
+ entry->plan = NULL;
+ }
+ }
+ dlist_insert_after(&plan_cache_lru.head, &entry->lru); /* prepend entry to the head of LRU list */
+ MemoryContextSwitchTo(old_context); /* Done with plan_cache_context memory context */
+
+
+ /*
+ * Prepare query only when it is executed more than autoprepare_threshold times
+ */
+ if (entry->disable_autoprepare || entry->exec_count++ < autoprepare_threshold)
+ {
+ undo_query_plan_changes(binding_list);
+ autoprepare_misses += 1;
+ return false;
+ }
+
+ if (entry->plan == NULL)
+ {
+ bool snapshot_set = false;
+ const char *commandTag;
+ List *querytree_list;
+
+ /*
+ * Switch to appropriate context for preparing plan.
+ */
+ old_context = MemoryContextSwitchTo(MessageContext);
+
+ /*
+ * Get the command name for use in status display (it also becomes the
+ * default completion tag, down inside PortalRun). Set ps_status and
+ * do any special start-of-SQL-command processing needed by the
+ * destination.
+ */
+ commandTag = CreateCommandTag(raw_parse_tree->stmt);
+
+ /*
+ * If we are in an aborted transaction, reject all commands except
+ * COMMIT/ABORT. It is important that this test occur before we try
+ * to do parse analysis, rewrite, or planning, since all those phases
+ * try to do database accesses, which may fail in abort state. (It
+ * might be safe to allow some additional utility commands in this
+ * state, but not many...)
+ */
+ if (IsAbortedTransactionBlockState() &&
+ !IsTransactionExitStmt(raw_parse_tree->stmt))
+ ereport(ERROR,
+ (errcode(ERRCODE_IN_FAILED_SQL_TRANSACTION),
+ errmsg("current transaction is aborted, "
+ "commands ignored until end of transaction block"),
+ errdetail_abort()));
+
+ /*
+ * Create the CachedPlanSource before we do parse analysis, since it
+ * needs to see the unmodified raw parse tree.
+ */
+ psrc = CreateCachedPlan(raw_parse_tree, query_string, commandTag);
+
+ /*
+ * Revert raw plan to use literals
+ */
+ undo_query_plan_changes(binding_list);
+
+ /*
+ * Set up a snapshot if parse analysis/planning will need one.
+ */
+ if (analyze_requires_snapshot(raw_parse_tree))
+ {
+ PushActiveSnapshot(GetTransactionSnapshot());
+ snapshot_set = true;
+ }
+
+ querytree_list = pg_analyze_and_rewrite(raw_parse_tree, query_string,
+ NULL, 0, NULL);
+ /*
+ * Replace Const with Param nodes
+ */
+ (void)query_tree_mutator((Query*)linitial(querytree_list),
+ parse_tree_generalizer,
+ binding_list,
+ QTW_DONT_COPY_QUERY);
+
+ /* Done with the snapshot used for parsing/planning */
+ if (snapshot_set)
+ PopActiveSnapshot();
+
+ param_types = (Oid*)palloc(sizeof(Oid)*n_params);
+ psrc->param_types = param_types;
+ for (paramno = 0, binding = binding_list; paramno < n_params; paramno++, binding = binding->next)
+ {
+ if (binding->param == NULL || binding->type == UNKNOWNOID)
+ {
+ /* Failed to resolve parameter type */
+ entry->disable_autoprepare = true;
+ autoprepare_misses += 1;
+ MemoryContextSwitchTo(old_context);
+ return false;
+ }
+ param_types[paramno] = binding->type;
+ }
+
+ /* Finish filling in the CachedPlanSource */
+ CompleteCachedPlan(psrc,
+ querytree_list,
+ NULL,
+ param_types,
+ n_params,
+ NULL,
+ NULL,
+ CURSOR_OPT_PARALLEL_OK, /* allow parallel mode */
+ true); /* fixed result */
+
+ /* If we got a cancel signal during analysis, quit */
+ CHECK_FOR_INTERRUPTS();
+
+ SaveCachedPlan(psrc);
+
+ /*
+ * We do NOT close the open transaction command here; that only happens
+ * when the client sends Sync. Instead, do CommandCounterIncrement just
+ * in case something happened during parse/plan.
+ */
+ CommandCounterIncrement();
+
+ MemoryContextSwitchTo(old_context); /* Done with MessageContext memory context */
+
+ entry->plan = psrc;
+
+ /*
+ * Determine output format
+ */
+ entry->format = 0; /* TEXT is default */
+ if (IsA(raw_parse_tree->stmt, FetchStmt))
+ {
+ FetchStmt *stmt = (FetchStmt *)raw_parse_tree->stmt;
+
+ if (!stmt->ismove)
+ {
+ Portal fportal = GetPortalByName(stmt->portalname);
+
+ if (PortalIsValid(fportal) &&
+ (fportal->cursorOptions & CURSOR_OPT_BINARY))
+ entry->format = 1; /* BINARY */
+ }
+ }
+ }
+ else
+ {
+ /* Plan found */
+ psrc = entry->plan;
+ Assert(n_params == entry->n_params);
+ }
+
+ /*
+ * If we are in aborted transaction state, the only portals we can
+ * actually run are those containing COMMIT or ROLLBACK commands. We
+ * disallow binding anything else to avoid problems with infrastructure
+ * that expects to run inside a valid transaction. We also disallow
+ * binding any parameters, since we can't risk calling user-defined I/O
+ * functions.
+ */
+ if (IsAbortedTransactionBlockState() &&
+ (!IsTransactionExitStmt(psrc->raw_parse_tree->stmt) ||
+ n_params != 0))
+ ereport(ERROR,
+ (errcode(ERRCODE_IN_FAILED_SQL_TRANSACTION),
+ errmsg("current transaction is aborted, "
+ "commands ignored until end of transaction block"),
+ errdetail_abort()));
+
+ /*
+ * Create unnamed portal to run the query or queries in. If there
+ * already is one, silently drop it.
+ */
+ portal = CreatePortal("", true, true);
+ /* Don't display the portal in pg_cursors */
+ portal->visible = false;
+
+ /*
+ * Prepare to copy stuff into the portal's memory context. We do all this
+ * copying first, because it could possibly fail (out-of-memory) and we
+ * don't want a failure to occur between GetCachedPlan and
+ * PortalDefineQuery; that would result in leaking our plancache refcount.
+ */
+ old_context = MemoryContextSwitchTo(PortalGetHeapMemory(portal));
+
+ /* Copy the plan's query string into the portal */
+ query_string = pstrdup(psrc->query_string);
+
+ /*
+ * Set a snapshot if we have parameters to fetch (since the input
+ * functions might need it) or the query isn't a utility command (and
+ * hence could require redoing parse analysis and planning). We keep the
+ * snapshot active till we're done, so that plancache.c doesn't have to
+ * take new ones.
+ */
+ if (n_params > 0 ||
+ (psrc->raw_parse_tree &&
+ analyze_requires_snapshot(psrc->raw_parse_tree)))
+ {
+ PushActiveSnapshot(GetTransactionSnapshot());
+ snapshot_set = true;
+ }
+
+ /*
+ * Fetch parameters, if any, and store in the portal's memory context.
+ */
+ if (n_params > 0)
+ {
+ ErrorContextCallback errcallback;
+
+ params = (ParamListInfo) palloc0(offsetof(ParamListInfoData, params) +
+ n_params * sizeof(ParamExternData));
+ params->numParams = n_params;
+
+ /*
+ * Register error callback to precisely report error in case of conversion error while storig parameter value.
+ */
+ errcallback.callback = prepare_error_callback;
+ errcallback.arg = (void *) psrc;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ for (paramno = 0, binding = binding_list;
+ paramno < n_params;
+ paramno++, binding = binding->next)
+ {
+ Oid ptype = psrc->param_types[paramno];
+
+ param_location = binding->literal->location;
+
+ params->params[paramno].isnull = false;
+ params->params[paramno].value = get_param_value(ptype, &binding->literal->val);
+ /*
+ * We mark the params as CONST. This ensures that any custom plan
+ * makes full use of the parameter values.
+ */
+ params->params[paramno].pflags = PARAM_FLAG_CONST;
+ params->params[paramno].ptype = ptype;
+ }
+ error_context_stack = errcallback.previous;
+ }
+ else
+ {
+ params = NULL;
+ }
+
+ /* Done storing stuff in portal's context */
+ MemoryContextSwitchTo(old_context);
+
+ /*
+ * Obtain a plan from the CachedPlanSource. Any cruft from (re)planning
+ * will be generated in MessageContext. The plan refcount will be
+ * assigned to the Portal, so it will be released at portal destruction.
+ */
+ cplan = GetCachedPlan(psrc, params, false, NULL);
+
+ /*
+ * Now we can define the portal.
+ *
+ * DO NOT put any code that could possibly throw an error between the
+ * above GetCachedPlan call and here.
+ */
+ PortalDefineQuery(portal,
+ NULL,
+ query_string,
+ psrc->commandTag,
+ cplan->stmt_list,
+ cplan);
+
+ /* Done with the snapshot used for parameter I/O and parsing/planning */
+ if (snapshot_set)
+ {
+ PopActiveSnapshot();
+ }
+
+ /*
+ * And we're ready to start portal execution.
+ */
+ PortalStart(portal, params, 0, InvalidSnapshot);
+
+ /*
+ * Apply the result format requests to the portal.
+ */
+ PortalSetResultFormat(portal, 1, &entry->format);
+
+ /*
+ * Finally execute prepared statement
+ */
+ exec_prepared_plan(portal, "", FETCH_ALL, whereToSendOutput);
+
+ /*
+ * Close down transaction statement, if one is open.
+ */
+ finish_xact_command();
+
+ autoprepare_hits += 1;
+
+ return true;
+}
+
+
+void ResetAutoprepareCache(void)
+{
+ if (plan_cache_hash != NULL)
+ {
+ hash_destroy(plan_cache_hash);
+ MemoryContextReset(plan_cache_context);
+ dlist_init(&plan_cache_lru);
+ autoprepare_cached_plans = 0;
+ plan_cache_hash = 0;
+ }
+}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 775477c..94ee570 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -684,10 +684,12 @@ standard_ProcessUtility(PlannedStmt *pstmt,
case T_AlterSystemStmt:
PreventTransactionChain(isTopLevel, "ALTER SYSTEM");
AlterSystemSetConfigFile((AlterSystemStmt *) parsetree);
+ ResetAutoprepareCache();
break;
case T_VariableSetStmt:
ExecSetVariableStmt((VariableSetStmt *) parsetree, isTopLevel);
+ ResetAutoprepareCache();
break;
case T_VariableShowStmt:
@@ -960,6 +962,8 @@ ProcessUtilitySlow(ParseState *pstate,
/* All event trigger calls are done only when isCompleteQuery is true */
needCleanup = isCompleteQuery && EventTriggerBeginCompleteQuery();
+ ResetAutoprepareCache();
+
/* PG_TRY block is to ensure we call EventTriggerEndCompleteQuery */
PG_TRY();
{
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index d0e54b8..af641f0 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -111,6 +111,7 @@
#include "utils/relmapper.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+#include "tcop/pquery.h"
/*
@@ -644,6 +645,7 @@ InvalidateSystemCaches(void)
InvalidateCatalogSnapshot();
ResetCatalogCaches();
+ ResetAutoprepareCache();
RelationCacheInvalidate(); /* gets smgr and relmap too */
for (i = 0; i < syscache_callback_count; i++)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 246fea8..a991c6f 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -472,6 +472,10 @@ int tcp_keepalives_idle;
int tcp_keepalives_interval;
int tcp_keepalives_count;
+
+int autoprepare_threshold;
+int autoprepare_limit;
+
/*
* SSL renegotiation was been removed in PostgreSQL 9.5, but we tolerate it
* being set to zero (meaning never renegotiate) for backward compatibility.
@@ -1958,6 +1962,28 @@ static struct config_int ConfigureNamesInt[] =
check_max_stack_depth, assign_max_stack_depth, NULL
},
+ /*
+ * Threshold for implicit preparing of frequently executed queries
+ */
+ {
+ {"autoprepare_threshold", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Threshold for autopreparing query."),
+ gettext_noop("0 value disables autoprepare.")
+ },
+ &autoprepare_threshold,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+ {
+ {"autoprepare_limit", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Maximal number of autoprepared queries."),
+ gettext_noop("0 means unlimited number of autoprepared queries. Too large number of prepared queries can cause backend memory overflow and slowdown execution speed (because of increased lookup time)")
+ },
+ &autoprepare_limit,
+ 113, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"temp_file_limit", PGC_SUSET, RESOURCES_DISK,
gettext_noop("Limits the total size of all temporary files used by each process."),
diff --git a/src/include/nodes/nodeFuncs.h b/src/include/nodes/nodeFuncs.h
index 3366983..263584d 100644
--- a/src/include/nodes/nodeFuncs.h
+++ b/src/include/nodes/nodeFuncs.h
@@ -73,6 +73,9 @@ extern Node *query_or_expression_tree_mutator(Node *node, Node *(*mutator) (),
extern bool raw_expression_tree_walker(Node *node, bool (*walker) (),
void *context);
+extern bool raw_expression_tree_mutator(Node *node, bool (*mutator) (),
+ void *context);
+
struct PlanState;
extern bool planstate_tree_walker(struct PlanState *planstate, bool (*walker) (),
void *context);
diff --git a/src/include/tcop/pquery.h b/src/include/tcop/pquery.h
index 6abfe7b..d25e744 100644
--- a/src/include/tcop/pquery.h
+++ b/src/include/tcop/pquery.h
@@ -42,4 +42,6 @@ extern uint64 PortalRunFetch(Portal portal,
long count,
DestReceiver *dest);
+extern void ResetAutoprepareCache(void);
+
#endif /* PQUERY_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index c1870d2..cd6a645 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -251,6 +251,9 @@ extern int client_min_messages;
extern int log_min_duration_statement;
extern int log_temp_files;
+extern int autoprepare_threshold;
+extern int autoprepare_limit;
+
extern int temp_file_limit;
extern int num_temp_buffers;
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Cached plans and statement generalization
@ 2017-09-12 17:11 Konstantin Knizhnik <[email protected]>
parent: Konstantin Knizhnik <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Konstantin Knizhnik @ 2017-09-12 17:11 UTC (permalink / raw)
To: pgsql-hackers
On 11.09.2017 12:24, Konstantin Knizhnik wrote:
> Attached please find rebased version of the patch.
> There are the updated performance results (pgbench -s 100 -c 1):
>
> protocol (-M)
> read-write
> read-only (-S)
> simple
> 3327
> 19325
> extended
> 2256
> 16908
> prepared
> 6145
> 39326
> simple+autoprepare
> 4950
> 34841
>
>
One more patch passing all regression tests with autoprepare_threshold=1.
I still do not think that it should be switch on by default...
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-patch] autoprepare-4.patch (71.6K, ../../[email protected]/3-autoprepare-4.patch)
download | inline diff:
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index e3eb0c5..17f3dfd 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -3688,6 +3688,454 @@ raw_expression_tree_walker(Node *node,
}
/*
+ * raw_expression_tree_mutator --- transform raw parse tree.
+ *
+ * This function is implementing slightly different approach for tree update than expression_tree_mutator().
+ * Callback is given pointer to pointer to the current node and can update this field instead of returning reference to new node.
+ * It makes it possible to remember changes and easily revert them without extra traversal of the tree.
+ *
+ * This function do not need QTW_DONT_COPY_QUERY flag: it never implicitly copy tree nodes, doing in-place update.
+ *
+ * Like raw_expression_tree_walker, there is no special rule about query
+ * boundaries: we descend to everything that's possibly interesting.
+ *
+ * Currently, the node type coverage here extends only to DML statements
+ * (SELECT/INSERT/UPDATE/DELETE) and nodes that can appear in them, because
+ * this is used mainly during analysis of CTEs, and only DML statements can
+ * appear in CTEs. If some other node is visited, iteration is immediately stopped and true is returned.
+ */
+bool
+raw_expression_tree_mutator(Node *node,
+ bool (*mutator) (),
+ void *context)
+{
+ ListCell *temp;
+
+ /*
+ * The walker has already visited the current node, and so we need only
+ * recurse into any sub-nodes it has.
+ */
+ if (node == NULL)
+ return false;
+
+ /* Guard against stack overflow due to overly complex expressions */
+ check_stack_depth();
+
+ switch (nodeTag(node))
+ {
+ case T_SetToDefault:
+ case T_CurrentOfExpr:
+ case T_Integer:
+ case T_Float:
+ case T_String:
+ case T_BitString:
+ case T_Null:
+ case T_ParamRef:
+ case T_A_Const:
+ case T_A_Star:
+ /* primitive node types with no subnodes */
+ break;
+ case T_Alias:
+ /* we assume the colnames list isn't interesting */
+ break;
+ case T_RangeVar:
+ return mutator(&((RangeVar *) node)->alias, context);
+ case T_GroupingFunc:
+ return mutator(&((GroupingFunc *) node)->args, context);
+ case T_SubLink:
+ {
+ SubLink *sublink = (SubLink *) node;
+
+ if (mutator(&sublink->testexpr, context))
+ return true;
+ /* we assume the operName is not interesting */
+ if (mutator(&sublink->subselect, context))
+ return true;
+ }
+ break;
+ case T_CaseExpr:
+ {
+ CaseExpr *caseexpr = (CaseExpr *) node;
+
+ if (mutator(&caseexpr->arg, context))
+ return true;
+ /* we assume mutator(& doesn't care about CaseWhens, either */
+ foreach(temp, caseexpr->args)
+ {
+ CaseWhen *when = (CaseWhen *) lfirst(temp);
+
+ Assert(IsA(when, CaseWhen));
+ if (mutator(&when->expr, context))
+ return true;
+ if (mutator(&when->result, context))
+ return true;
+ }
+ if (mutator(&caseexpr->defresult, context))
+ return true;
+ }
+ break;
+ case T_RowExpr:
+ /* Assume colnames isn't interesting */
+ return mutator(&((RowExpr *) node)->args, context);
+ case T_CoalesceExpr:
+ return mutator(&((CoalesceExpr *) node)->args, context);
+ case T_MinMaxExpr:
+ return mutator(&((MinMaxExpr *) node)->args, context);
+ case T_XmlExpr:
+ {
+ XmlExpr *xexpr = (XmlExpr *) node;
+
+ if (mutator(&xexpr->named_args, context))
+ return true;
+ /* we assume mutator(& doesn't care about arg_names */
+ if (mutator(&xexpr->args, context))
+ return true;
+ }
+ break;
+ case T_NullTest:
+ return mutator(&((NullTest *) node)->arg, context);
+ case T_BooleanTest:
+ return mutator(&((BooleanTest *) node)->arg, context);
+ case T_JoinExpr:
+ {
+ JoinExpr *join = (JoinExpr *) node;
+
+ if (mutator(&join->larg, context))
+ return true;
+ if (mutator(&join->rarg, context))
+ return true;
+ if (mutator(&join->quals, context))
+ return true;
+ if (mutator(&join->alias, context))
+ return true;
+ /* using list is deemed uninteresting */
+ }
+ break;
+ case T_IntoClause:
+ {
+ IntoClause *into = (IntoClause *) node;
+
+ if (mutator(&into->rel, context))
+ return true;
+ /* colNames, options are deemed uninteresting */
+ /* viewQuery should be null in raw parsetree, but check it */
+ if (mutator(&into->viewQuery, context))
+ return true;
+ }
+ break;
+ case T_List:
+ foreach(temp, (List *) node)
+ {
+ if (mutator(&lfirst(temp), context))
+ return true;
+ }
+ break;
+ case T_InsertStmt:
+ {
+ InsertStmt *stmt = (InsertStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->cols, context))
+ return true;
+ if (mutator(&stmt->selectStmt, context))
+ return true;
+ if (mutator(&stmt->onConflictClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_DeleteStmt:
+ {
+ DeleteStmt *stmt = (DeleteStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->usingClause, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_UpdateStmt:
+ {
+ UpdateStmt *stmt = (UpdateStmt *) node;
+
+ if (mutator(&stmt->relation, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->fromClause, context))
+ return true;
+ if (mutator(&stmt->returningList, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ }
+ break;
+ case T_SelectStmt:
+ {
+ SelectStmt *stmt = (SelectStmt *) node;
+
+ if (mutator(&stmt->distinctClause, context))
+ return true;
+ if (mutator(&stmt->intoClause, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->fromClause, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ if (mutator(&stmt->groupClause, context))
+ return true;
+ if (mutator(&stmt->havingClause, context))
+ return true;
+ if (mutator(&stmt->windowClause, context))
+ return true;
+ if (mutator(&stmt->valuesLists, context))
+ return true;
+ if (mutator(&stmt->sortClause, context))
+ return true;
+ if (mutator(&stmt->limitOffset, context))
+ return true;
+ if (mutator(&stmt->limitCount, context))
+ return true;
+ if (mutator(&stmt->lockingClause, context))
+ return true;
+ if (mutator(&stmt->withClause, context))
+ return true;
+ if (mutator(&stmt->larg, context))
+ return true;
+ if (mutator(&stmt->rarg, context))
+ return true;
+ }
+ break;
+ case T_A_Expr:
+ {
+ A_Expr *expr = (A_Expr *) node;
+
+ if (mutator(&expr->lexpr, context))
+ return true;
+ if (mutator(&expr->rexpr, context))
+ return true;
+ /* operator name is deemed uninteresting */
+ }
+ break;
+ case T_BoolExpr:
+ {
+ BoolExpr *expr = (BoolExpr *) node;
+
+ if (mutator(&expr->args, context))
+ return true;
+ }
+ break;
+ case T_ColumnRef:
+ /* we assume the fields contain nothing interesting */
+ break;
+ case T_FuncCall:
+ {
+ FuncCall *fcall = (FuncCall *) node;
+
+ if (mutator(&fcall->args, context))
+ return true;
+ if (mutator(&fcall->agg_order, context))
+ return true;
+ if (mutator(&fcall->agg_filter, context))
+ return true;
+ if (mutator(&fcall->over, context))
+ return true;
+ /* function name is deemed uninteresting */
+ }
+ break;
+ case T_NamedArgExpr:
+ return mutator(&((NamedArgExpr *) node)->arg, context);
+ case T_A_Indices:
+ {
+ A_Indices *indices = (A_Indices *) node;
+
+ if (mutator(&indices->lidx, context))
+ return true;
+ if (mutator(&indices->uidx, context))
+ return true;
+ }
+ break;
+ case T_A_Indirection:
+ {
+ A_Indirection *indir = (A_Indirection *) node;
+
+ if (mutator(&indir->arg, context))
+ return true;
+ if (mutator(&indir->indirection, context))
+ return true;
+ }
+ break;
+ case T_A_ArrayExpr:
+ return mutator(&((A_ArrayExpr *) node)->elements, context);
+ case T_ResTarget:
+ {
+ ResTarget *rt = (ResTarget *) node;
+
+ if (mutator(&rt->indirection, context))
+ return true;
+ if (mutator(&rt->val, context))
+ return true;
+ }
+ break;
+ case T_MultiAssignRef:
+ return mutator(&((MultiAssignRef *) node)->source, context);
+ case T_TypeCast:
+ {
+ TypeCast *tc = (TypeCast *) node;
+
+ if (mutator(&tc->arg, context))
+ return true;
+ if (mutator(&tc->typeName, context))
+ return true;
+ }
+ break;
+ case T_CollateClause:
+ return mutator(&((CollateClause *) node)->arg, context);
+ case T_SortBy:
+ return mutator(&((SortBy *) node)->node, context);
+ case T_WindowDef:
+ {
+ WindowDef *wd = (WindowDef *) node;
+
+ if (mutator(&wd->partitionClause, context))
+ return true;
+ if (mutator(&wd->orderClause, context))
+ return true;
+ if (mutator(&wd->startOffset, context))
+ return true;
+ if (mutator(&wd->endOffset, context))
+ return true;
+ }
+ break;
+ case T_RangeSubselect:
+ {
+ RangeSubselect *rs = (RangeSubselect *) node;
+
+ if (mutator(&rs->subquery, context))
+ return true;
+ if (mutator(&rs->alias, context))
+ return true;
+ }
+ break;
+ case T_RangeFunction:
+ {
+ RangeFunction *rf = (RangeFunction *) node;
+
+ if (mutator(&rf->functions, context))
+ return true;
+ if (mutator(&rf->alias, context))
+ return true;
+ if (mutator(&rf->coldeflist, context))
+ return true;
+ }
+ break;
+ case T_RangeTableSample:
+ {
+ RangeTableSample *rts = (RangeTableSample *) node;
+
+ if (mutator(&rts->relation, context))
+ return true;
+ /* method name is deemed uninteresting */
+ if (mutator(&rts->args, context))
+ return true;
+ if (mutator(&rts->repeatable, context))
+ return true;
+ }
+ break;
+ case T_TypeName:
+ {
+ TypeName *tn = (TypeName *) node;
+
+ if (mutator(&tn->typmods, context))
+ return true;
+ if (mutator(&tn->arrayBounds, context))
+ return true;
+ /* type name itself is deemed uninteresting */
+ }
+ break;
+ case T_ColumnDef:
+ {
+ ColumnDef *coldef = (ColumnDef *) node;
+
+ if (mutator(&coldef->typeName, context))
+ return true;
+ if (mutator(&coldef->raw_default, context))
+ return true;
+ if (mutator(&coldef->collClause, context))
+ return true;
+ /* for now, constraints are ignored */
+ }
+ break;
+ case T_IndexElem:
+ {
+ IndexElem *indelem = (IndexElem *) node;
+
+ if (mutator(&indelem->expr, context))
+ return true;
+ /* collation and opclass names are deemed uninteresting */
+ }
+ break;
+ case T_GroupingSet:
+ return mutator(&((GroupingSet *) node)->content, context);
+ case T_LockingClause:
+ return mutator(&((LockingClause *) node)->lockedRels, context);
+ case T_XmlSerialize:
+ {
+ XmlSerialize *xs = (XmlSerialize *) node;
+
+ if (mutator(&xs->expr, context))
+ return true;
+ if (mutator(&xs->typeName, context))
+ return true;
+ }
+ break;
+ case T_WithClause:
+ return mutator(&((WithClause *) node)->ctes, context);
+ case T_InferClause:
+ {
+ InferClause *stmt = (InferClause *) node;
+
+ if (mutator(&stmt->indexElems, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ }
+ break;
+ case T_OnConflictClause:
+ {
+ OnConflictClause *stmt = (OnConflictClause *) node;
+
+ if (mutator(&stmt->infer, context))
+ return true;
+ if (mutator(&stmt->targetList, context))
+ return true;
+ if (mutator(&stmt->whereClause, context))
+ return true;
+ }
+ break;
+ case T_CommonTableExpr:
+ return mutator(&((CommonTableExpr *) node)->ctequery, context);
+ default:
+ return true;
+ }
+ return false;
+}
+
+/*
* planstate_tree_walker --- walk plan state trees
*
* The walker has already visited the current node, and so we need only
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index c10d891..e0d62ef 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -42,11 +42,13 @@
#include "catalog/pg_type.h"
#include "commands/async.h"
#include "commands/prepare.h"
+#include "commands/defrem.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "libpq/pqsignal.h"
#include "miscadmin.h"
#include "nodes/print.h"
+#include "nodes/nodeFuncs.h"
#include "optimizer/planner.h"
#include "pgstat.h"
#include "pg_trace.h"
@@ -75,6 +77,7 @@
#include "utils/snapmgr.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
+#include "utils/int8.h"
#include "mb/pg_wchar.h"
@@ -182,7 +185,8 @@ static bool IsTransactionExitStmtList(List *pstmts);
static bool IsTransactionStmtList(List *pstmts);
static void drop_unnamed_stmt(void);
static void log_disconnections(int code, Datum arg);
-
+static bool exec_cached_query(const char* query, List *parsetree_list);
+static void exec_prepared_plan(Portal portal, const char *portal_name, long max_rows, CommandDest dest);
/* ----------------------------------------------------------------
* routines to obtain user input
@@ -956,6 +960,16 @@ exec_simple_query(const char *query_string)
use_implicit_block = (list_length(parsetree_list) > 1);
/*
+ * Try to find cached plan.
+ */
+ if (autoprepare_threshold != 0
+ && list_length(parsetree_list) == 1 /* we can prepare only single statement commands */
+ && exec_cached_query(query_string, parsetree_list))
+ {
+ return;
+ }
+
+ /*
* Run through the raw parsetree(s) and process each one.
*/
foreach(parsetree_item, parsetree_list)
@@ -1852,9 +1866,28 @@ exec_bind_message(StringInfo input_message)
static void
exec_execute_message(const char *portal_name, long max_rows)
{
- CommandDest dest;
+ Portal portal = GetPortalByName(portal_name);
+ CommandDest dest = whereToSendOutput;
+
+ /* Adjust destination to tell printtup.c what to do */
+ if (dest == DestRemote)
+ dest = DestRemoteExecute;
+
+ if (!PortalIsValid(portal))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_CURSOR),
+ errmsg("portal \"%s\" does not exist", portal_name)));
+
+ exec_prepared_plan(portal, portal_name, max_rows, dest);
+}
+
+/*
+ * Execute prepared plan.
+ */
+static void
+exec_prepared_plan(Portal portal, const char *portal_name, long max_rows, CommandDest dest)
+{
DestReceiver *receiver;
- Portal portal;
bool completed;
char completionTag[COMPLETION_TAG_BUFSIZE];
const char *sourceText;
@@ -1866,17 +1899,6 @@ exec_execute_message(const char *portal_name, long max_rows)
bool was_logged = false;
char msec_str[32];
- /* Adjust destination to tell printtup.c what to do */
- dest = whereToSendOutput;
- if (dest == DestRemote)
- dest = DestRemoteExecute;
-
- portal = GetPortalByName(portal_name);
- if (!PortalIsValid(portal))
- ereport(ERROR,
- (errcode(ERRCODE_UNDEFINED_CURSOR),
- errmsg("portal \"%s\" does not exist", portal_name)));
-
/*
* If the original query was a null string, just return
* EmptyQueryResponse.
@@ -1939,7 +1961,7 @@ exec_execute_message(const char *portal_name, long max_rows)
* context, because that may get deleted if portal contains VACUUM).
*/
receiver = CreateDestReceiver(dest);
- if (dest == DestRemoteExecute)
+ if (dest == DestRemoteExecute || dest == DestRemote)
SetRemoteDestReceiverParams(receiver, portal);
/*
@@ -4534,3 +4556,782 @@ log_disconnections(int code, Datum arg)
port->user_name, port->database_name, port->remote_host,
port->remote_port[0] ? " port=" : "", port->remote_port)));
}
+
+/*
+ * Autoprepare implementation.
+ * Autoprepare consists of raw parse tree mutator, hash table of cached plans and exec_cached_query function
+ * which combines exec_parse_message + exec_bind_message + exec_execute_message
+ */
+
+/*
+ * Mapping between parameters and replaced literals
+ */
+typedef struct ParamBinding
+{
+ A_Const* literal; /* Original literal */
+ ParamRef* paramref;/* Constructed parameter reference */
+ Param* param; /* Constructed parameter */
+ Node** ref; /* Pointer to pointer to literal node (used to revert raw parse tree update) */
+ Oid raw_type;/* Parameter raw type */
+ Oid type; /* Parameter type after analysis */
+ struct ParamBinding* next; /* L1-list of query parameter bindings */
+} ParamBinding;
+
+/*
+ * Plan cache entry
+ */
+typedef struct
+{
+ Node* parse_tree; /* tree is used as hash key */
+ dlist_node lru; /* double linked list to implement LRU */
+ int64 exec_count; /* counter of execution of this query */
+ CachedPlanSource* plan;
+ uint32 hash; /* hash calculated for this parsed tree */
+ Oid* param_types;/* types of parameters */
+ int n_params; /* number of parameters extracted for this query */
+ int16 format; /* portal output format */
+ bool disable_autoprepare; /* disable preparing of this query */
+} plan_cache_entry;
+
+static uint32 plan_cache_hash_fn(const void *key, Size keysize)
+{
+ return ((plan_cache_entry*)key)->hash;
+}
+
+static int plan_cache_match_fn(const void *key1, const void *key2, Size keysize)
+{
+ plan_cache_entry* e1 = (plan_cache_entry*)key1;
+ plan_cache_entry* e2 = (plan_cache_entry*)key2;
+
+ return equal(e1->parse_tree, e2->parse_tree)
+ && memcmp(e1->param_types, e2->param_types, sizeof(Oid)*e1->n_params) == 0 ? 0 : 1;
+}
+
+static void* plan_cache_keycopy_fn(void *dest, const void *src, Size keysize)
+{
+ plan_cache_entry* dst_entry = (plan_cache_entry*)dest;
+ plan_cache_entry* src_entry = (plan_cache_entry*)src;
+ dst_entry->parse_tree = copyObject(src_entry->parse_tree);
+ dst_entry->param_types = palloc(src_entry->n_params*sizeof(Oid));
+ dst_entry->n_params = src_entry->n_params;
+ memcpy(dst_entry->param_types, src_entry->param_types, src_entry->n_params*sizeof(Oid));
+ dst_entry->hash = src_entry->hash;
+ return dest;
+}
+
+#define PLAN_CACHE_SIZE 113
+
+/*
+ * Plan cache access statistic
+ */
+size_t autoprepare_hits;
+size_t autoprepare_misses;
+size_t autoprepare_cached_plans;
+
+/*
+ * Context for raw_expression_tree_mutator
+ */
+typedef struct {
+ int n_params; /* Number of extracted parameters */
+ uint32 hash; /* We calculate hash for parse tree during plan traversal */
+ ParamBinding** param_list_tail; /* pointer to last element "next" field address, used to construct L1 list of parameters */
+} GeneralizerCtx;
+
+
+static HTAB* plan_cache_hash; /* hash table for plan cache */
+static dlist_head plan_cache_lru; /* LRU L2-list for cached queries */
+static MemoryContext plan_cache_context; /* memory context used for plan cache */
+
+/*
+ * Check if expression is constant (used to eliminate substitution of literals with parameters in such expressions
+ */
+static bool is_constant_expression(Node* node)
+{
+ return node != NULL
+ && (IsA(node, A_Const)
+ || (IsA(node, A_Expr)
+ && is_constant_expression(((A_Expr*)node)->lexpr)
+ && is_constant_expression(((A_Expr*)node)->rexpr)));
+}
+
+/*
+ * Infer type of literal expression. Null literals should not be replaced with parameters.
+ */
+static Oid get_literal_type(Value* val)
+{
+ int64 val64;
+ switch (val->type)
+ {
+ case T_Integer:
+ return INT4OID;
+ case T_Float:
+ /* could be an oversize integer as well as a float ... */
+ if (scanint8(strVal(val), true, &val64))
+ {
+ /*
+ * It might actually fit in int32. Probably only INT_MIN can
+ * occur, but we'll code the test generally just to be sure.
+ */
+ int32 val32 = (int32) val64;
+ return (val64 == (int64)val32) ? INT4OID : INT8OID;
+ }
+ else
+ {
+ return NUMERICOID;
+ }
+ case T_BitString:
+ return BITOID;
+ case T_String:
+ return UNKNOWNOID;
+ default:
+ Assert(false);
+ return InvalidOid;
+ }
+}
+
+static Datum get_param_value(Oid type, Value* val)
+{
+ if (val->type == T_Integer && type == INT4OID)
+ {
+ /*
+ * Integer constant
+ */
+ return Int32GetDatum((int32)val->val.ival);
+ }
+ else
+ {
+ /*
+ * Convert from string literal
+ */
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(type, &typinput, &typioparam);
+ return OidInputFunctionCall(typinput, val->val.str, typioparam, -1);
+ }
+}
+
+
+/*
+ * Callback for raw_expression_tree_mutator performing substitution of literals with parameters
+ */
+static bool
+raw_parse_tree_generalizer(Node** ref, void *context)
+{
+ Node* node = *ref;
+ GeneralizerCtx* ctx = (GeneralizerCtx*)context;
+ if (node == NULL)
+ {
+ return false;
+ }
+ /*
+ * Calculate hash for parse tree. We consider only node tags here, precise comparison of trees is done using equal() function.
+ * Here we calculate hash for original (unpatched) tree, without ParamRef nodes.
+ * It is non principle, because hash calculation doesn't take in account types and values of Const nodes. So the same generalized queries
+ * will have the same hash value. There are about 1000 different nodes tags, this is why we rotate hash on 10 bits.
+ */
+ ctx->hash = (ctx->hash << 10) ^ (ctx->hash >> 22) ^ nodeTag(node);
+
+ switch (nodeTag(node))
+ {
+ case T_A_Expr:
+ {
+ /*
+ * Do not perform substitution of literals in constant expression (which is likely to be the same for all queries and optimized by compiler)
+ */
+ if (!is_constant_expression(node))
+ {
+ A_Expr *expr = (A_Expr *) node;
+ if (raw_parse_tree_generalizer((Node**)&expr->lexpr, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&expr->rexpr, context))
+ return true;
+ }
+ break;
+ }
+ case T_A_Const:
+ {
+ /*
+ * Do substitution of literals with parameters here
+ */
+ A_Const* literal = (A_Const*)node;
+ if (literal->val.type != T_Null)
+ {
+ /*
+ * Do not substitute null literals with parameters
+ */
+ ParamBinding* cp = palloc0(sizeof(ParamBinding));
+ ParamRef* param = makeNode(ParamRef);
+ param->number = ++ctx->n_params;
+ param->location = literal->location;
+ cp->ref = ref;
+ cp->paramref = param;
+ cp->literal = literal;
+ cp->raw_type = get_literal_type(&literal->val);
+ *ctx->param_list_tail = cp;
+ ctx->param_list_tail = &cp->next;
+ *ref = (Node*)param;
+ }
+ break;
+ }
+ case T_SelectStmt:
+ {
+ /*
+ * Substitute literals only in target list, WHERE, VALUES and WITH clause,
+ * skipping target and from lists, which is unlikely contains some parameterized values
+ */
+ SelectStmt *stmt = (SelectStmt *) node;
+ if (stmt->intoClause)
+ return true; /* Utility statement can not be prepared */
+ if (raw_parse_tree_generalizer((Node**)&stmt->targetList, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->whereClause, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->valuesLists, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->withClause, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->larg, context))
+ return true;
+ if (raw_parse_tree_generalizer((Node**)&stmt->rarg, context))
+ return true;
+ break;
+ }
+ case T_TypeName:
+ case T_SortGroupClause:
+ case T_SortBy:
+ case T_A_ArrayExpr:
+ case T_TypeCast:
+ /*
+ * Literals in this clauses should not be replaced with parameters
+ */
+ break;
+ default:
+ /*
+ * Default traversal. raw_expression_tree_mutator returns true for all not recognized nodes, for example right now
+ * all transaction control statements are not covered by raw_expression_tree_mutator and so will not auto prepared.
+ * My experiments show that effect of non-preparing start/commit transaction statements is positive.
+ */
+ return raw_expression_tree_mutator(node, raw_parse_tree_generalizer, context);
+ }
+ return false;
+}
+
+static Node*
+parse_tree_generalizer(Node *node, void *context)
+{
+ ParamBinding* binding;
+ ParamBinding* binding_list = (ParamBinding*)context;
+ if (node == NULL)
+ {
+ return NULL;
+ }
+ if (IsA(node, Query))
+ {
+ return (Node*)query_tree_mutator((Query*)node,
+ parse_tree_generalizer,
+ context,
+ QTW_DONT_COPY_QUERY);
+ }
+ if (IsA(node, Const))
+ {
+ Const* c = (Const*)node;
+ int paramno = 1;
+ for (binding = binding_list; binding != NULL && binding->literal->location != c->location; binding = binding->next, paramno++);
+ if (binding != NULL)
+ {
+ if (binding->param != NULL)
+ {
+ /* Parameter can be used only once */
+ binding->type = UNKNOWNOID;
+ //return (Node*)binding->param;
+ }
+ else
+ {
+ Param* param = makeNode(Param);
+ param->paramkind = PARAM_EXTERN;
+ param->paramid = paramno;
+ param->paramtype = c->consttype;
+ param->paramtypmod = c->consttypmod;
+ param->paramcollid = c->constcollid;
+ param->location = c->location;
+ binding->type = c->consttype;
+ binding->param = param;
+ return (Node*)param;
+ }
+ }
+ return node;
+ }
+ return expression_tree_mutator(node, parse_tree_generalizer, context);
+}
+
+/*
+ * Restore original parse tree, replacing all ParamRef back with Const nodes.
+ * Such undo operation seems to be more efficient than copying the whole parse tree by raw_expression_tree_mutator
+ */
+static void undo_query_plan_changes(ParamBinding* cp)
+{
+ while (cp != NULL) {
+ *cp->ref = (Node*)cp->literal;
+ cp = cp->next;
+ }
+}
+
+/*
+ * Callback for raw_expression_tree_walker dropping parse tree
+ */
+static bool drop_tree_node(Node* node, void* context)
+{
+ if (node) {
+ raw_expression_tree_walker(node, drop_tree_node, NULL);
+ pfree(node);
+ }
+ return false;
+}
+
+/*
+ * Location of converted literal in query.
+ * Used for precise error reporting (line number)
+ */
+static int param_location;
+
+/*
+ * Error callback adding information about error location
+ */
+static void
+prepare_error_callback(void *arg)
+{
+ CachedPlanSource *psrc = (CachedPlanSource*)arg;
+ /* And pass it to the ereport mechanism */
+ if (geterrcode() != ERRCODE_QUERY_CANCELED) {
+ int pos = pg_mbstrlen_with_len(psrc->query_string, param_location) + 1;
+ (void)errposition(pos);
+ }
+}
+/*
+ * Try to generalize query, find cached plan for it and execute
+ */
+static bool exec_cached_query(const char *query_string, List *parsetree_list)
+{
+ int n_params;
+ plan_cache_entry *entry;
+ bool found;
+ MemoryContext old_context;
+ CachedPlanSource *psrc;
+ ParamListInfo params;
+ int paramno;
+ CachedPlan *cplan;
+ Portal portal;
+ bool snapshot_set = false;
+ GeneralizerCtx ctx;
+ ParamBinding* binding;
+ ParamBinding* binding_list;
+ plan_cache_entry pattern;
+ Oid* param_types;
+ RawStmt *raw_parse_tree;
+
+ raw_parse_tree = castNode(RawStmt, linitial(parsetree_list));
+
+ /*
+ * Substitute literals with parameters and calculate hash for raw parse tree
+ */
+ ctx.param_list_tail = &binding_list;
+ ctx.n_params = 0;
+ ctx.hash = 0;
+ if (raw_parse_tree_generalizer(&raw_parse_tree->stmt, &ctx))
+ {
+ *ctx.param_list_tail = NULL;
+ undo_query_plan_changes(binding_list);
+ autoprepare_misses += 1;
+ return false;
+ }
+ *ctx.param_list_tail = NULL;
+ n_params = ctx.n_params;
+
+ /*
+ * Extract array of parameters types: it is needed for cached plan lookup
+ */
+ param_types = (Oid*)palloc(sizeof(Oid)*n_params);
+ for (paramno = 0, binding = binding_list; paramno < n_params; paramno++, binding = binding->next)
+ {
+ param_types[paramno] = binding->raw_type;
+ }
+
+ /*
+ * Construct plan cache context if not constructed yet.
+ */
+ if (plan_cache_context == NULL)
+ {
+ plan_cache_context = AllocSetContextCreate(TopMemoryContext,
+ "plan cache context",
+ ALLOCSET_DEFAULT_SIZES);
+ }
+ /* Manipulations with hash table are performed in plan_cache_context memory context */
+ old_context = MemoryContextSwitchTo(plan_cache_context);
+
+ /*
+ * Initialize hash table if not initialized yet
+ */
+ if (plan_cache_hash == NULL)
+ {
+ static HASHCTL info;
+ info.keysize = sizeof(plan_cache_entry);
+ info.entrysize = sizeof(plan_cache_entry);
+ info.hash = plan_cache_hash_fn;
+ info.match = plan_cache_match_fn;
+ info.keycopy = plan_cache_keycopy_fn;
+ plan_cache_hash = hash_create("plan_cache", autoprepare_limit != 0 ? autoprepare_limit : PLAN_CACHE_SIZE,
+ &info, HASH_ELEM | HASH_FUNCTION | HASH_COMPARE | HASH_KEYCOPY);
+ dlist_init(&plan_cache_lru);
+ }
+
+ /*
+ * Lookup generalized query
+ */
+ pattern.parse_tree = raw_parse_tree->stmt;
+ pattern.hash = ctx.hash;
+ pattern.n_params = n_params;
+ pattern.param_types = param_types;
+ entry = (plan_cache_entry*)hash_search(plan_cache_hash, &pattern, HASH_ENTER, &found);
+ if (!found)
+ {
+ /* Check number of cached queries */
+ if (++autoprepare_cached_plans > autoprepare_limit && autoprepare_limit != 0)
+ {
+ /* Drop least recently accessed query */
+ plan_cache_entry* victim = dlist_container(plan_cache_entry, lru, plan_cache_lru.head.prev);
+ Node* dropped_tree = victim->parse_tree;
+ dlist_delete(&victim->lru);
+ if (victim->plan)
+ {
+ DropCachedPlan(victim->plan);
+ }
+ pfree(victim->param_types);
+ hash_search(plan_cache_hash, victim, HASH_REMOVE, NULL);
+ raw_expression_tree_walker(dropped_tree, drop_tree_node, NULL);
+ autoprepare_cached_plans -= 1;
+ }
+ entry->exec_count = 0;
+ entry->plan = NULL;
+ entry->disable_autoprepare = false;
+ }
+ else
+ {
+ dlist_delete(&entry->lru); /* accessed entry will be moved to the head of LRU list */
+ if (entry->plan != NULL && !entry->plan->is_valid)
+ {
+ /* Drop invalidated plan: it will be reconstructed later */
+ DropCachedPlan(entry->plan);
+ entry->plan = NULL;
+ }
+ }
+ dlist_insert_after(&plan_cache_lru.head, &entry->lru); /* prepend entry to the head of LRU list */
+ MemoryContextSwitchTo(old_context); /* Done with plan_cache_context memory context */
+
+
+ /*
+ * Prepare query only when it is executed more than autoprepare_threshold times
+ */
+ if (entry->disable_autoprepare || entry->exec_count++ < autoprepare_threshold)
+ {
+ undo_query_plan_changes(binding_list);
+ autoprepare_misses += 1;
+ return false;
+ }
+
+ if (entry->plan == NULL)
+ {
+ bool snapshot_set = false;
+ const char *commandTag;
+ List *querytree_list;
+
+ /*
+ * Switch to appropriate context for preparing plan.
+ */
+ old_context = MemoryContextSwitchTo(MessageContext);
+
+ /*
+ * Get the command name for use in status display (it also becomes the
+ * default completion tag, down inside PortalRun). Set ps_status and
+ * do any special start-of-SQL-command processing needed by the
+ * destination.
+ */
+ commandTag = CreateCommandTag(raw_parse_tree->stmt);
+
+ /*
+ * If we are in an aborted transaction, reject all commands except
+ * COMMIT/ABORT. It is important that this test occur before we try
+ * to do parse analysis, rewrite, or planning, since all those phases
+ * try to do database accesses, which may fail in abort state. (It
+ * might be safe to allow some additional utility commands in this
+ * state, but not many...)
+ */
+ if (IsAbortedTransactionBlockState() &&
+ !IsTransactionExitStmt(raw_parse_tree->stmt))
+ ereport(ERROR,
+ (errcode(ERRCODE_IN_FAILED_SQL_TRANSACTION),
+ errmsg("current transaction is aborted, "
+ "commands ignored until end of transaction block"),
+ errdetail_abort()));
+
+ /*
+ * Create the CachedPlanSource before we do parse analysis, since it
+ * needs to see the unmodified raw parse tree.
+ */
+ psrc = CreateCachedPlan(raw_parse_tree, query_string, commandTag);
+
+ /*
+ * Revert raw plan to use literals
+ */
+ undo_query_plan_changes(binding_list);
+
+ /*
+ * Set up a snapshot if parse analysis/planning will need one.
+ */
+ if (analyze_requires_snapshot(raw_parse_tree))
+ {
+ PushActiveSnapshot(GetTransactionSnapshot());
+ snapshot_set = true;
+ }
+
+ querytree_list = pg_analyze_and_rewrite(raw_parse_tree, query_string,
+ NULL, 0, NULL);
+ /*
+ * Replace Const with Param nodes
+ */
+ (void)query_tree_mutator((Query*)linitial(querytree_list),
+ parse_tree_generalizer,
+ binding_list,
+ QTW_DONT_COPY_QUERY);
+
+ /* Done with the snapshot used for parsing/planning */
+ if (snapshot_set)
+ PopActiveSnapshot();
+
+ param_types = (Oid*)palloc(sizeof(Oid)*n_params);
+ psrc->param_types = param_types;
+ for (paramno = 0, binding = binding_list; paramno < n_params; paramno++, binding = binding->next)
+ {
+ if (binding->param == NULL || binding->type == UNKNOWNOID)
+ {
+ /* Failed to resolve parameter type */
+ entry->disable_autoprepare = true;
+ autoprepare_misses += 1;
+ MemoryContextSwitchTo(old_context);
+ return false;
+ }
+ param_types[paramno] = binding->type;
+ }
+
+ /* Finish filling in the CachedPlanSource */
+ CompleteCachedPlan(psrc,
+ querytree_list,
+ NULL,
+ param_types,
+ n_params,
+ NULL,
+ NULL,
+ CURSOR_OPT_PARALLEL_OK, /* allow parallel mode */
+ true); /* fixed result */
+
+ /* If we got a cancel signal during analysis, quit */
+ CHECK_FOR_INTERRUPTS();
+
+ SaveCachedPlan(psrc);
+
+ /*
+ * We do NOT close the open transaction command here; that only happens
+ * when the client sends Sync. Instead, do CommandCounterIncrement just
+ * in case something happened during parse/plan.
+ */
+ CommandCounterIncrement();
+
+ MemoryContextSwitchTo(old_context); /* Done with MessageContext memory context */
+
+ entry->plan = psrc;
+
+ /*
+ * Determine output format
+ */
+ entry->format = 0; /* TEXT is default */
+ if (IsA(raw_parse_tree->stmt, FetchStmt))
+ {
+ FetchStmt *stmt = (FetchStmt *)raw_parse_tree->stmt;
+
+ if (!stmt->ismove)
+ {
+ Portal fportal = GetPortalByName(stmt->portalname);
+
+ if (PortalIsValid(fportal) &&
+ (fportal->cursorOptions & CURSOR_OPT_BINARY))
+ entry->format = 1; /* BINARY */
+ }
+ }
+ }
+ else
+ {
+ /* Plan found */
+ psrc = entry->plan;
+ Assert(n_params == entry->n_params);
+ }
+
+ /*
+ * If we are in aborted transaction state, the only portals we can
+ * actually run are those containing COMMIT or ROLLBACK commands. We
+ * disallow binding anything else to avoid problems with infrastructure
+ * that expects to run inside a valid transaction. We also disallow
+ * binding any parameters, since we can't risk calling user-defined I/O
+ * functions.
+ */
+ if (IsAbortedTransactionBlockState() &&
+ (!IsTransactionExitStmt(psrc->raw_parse_tree->stmt) ||
+ n_params != 0))
+ ereport(ERROR,
+ (errcode(ERRCODE_IN_FAILED_SQL_TRANSACTION),
+ errmsg("current transaction is aborted, "
+ "commands ignored until end of transaction block"),
+ errdetail_abort()));
+
+ /*
+ * Create unnamed portal to run the query or queries in. If there
+ * already is one, silently drop it.
+ */
+ portal = CreatePortal("", true, true);
+ /* Don't display the portal in pg_cursors */
+ portal->visible = false;
+
+ /*
+ * Prepare to copy stuff into the portal's memory context. We do all this
+ * copying first, because it could possibly fail (out-of-memory) and we
+ * don't want a failure to occur between GetCachedPlan and
+ * PortalDefineQuery; that would result in leaking our plancache refcount.
+ */
+ old_context = MemoryContextSwitchTo(PortalGetHeapMemory(portal));
+
+ /* Copy the plan's query string into the portal */
+ query_string = pstrdup(psrc->query_string);
+
+ /*
+ * Set a snapshot if we have parameters to fetch (since the input
+ * functions might need it) or the query isn't a utility command (and
+ * hence could require redoing parse analysis and planning). We keep the
+ * snapshot active till we're done, so that plancache.c doesn't have to
+ * take new ones.
+ */
+ if (n_params > 0 ||
+ (psrc->raw_parse_tree &&
+ analyze_requires_snapshot(psrc->raw_parse_tree)))
+ {
+ PushActiveSnapshot(GetTransactionSnapshot());
+ snapshot_set = true;
+ }
+
+ /*
+ * Fetch parameters, if any, and store in the portal's memory context.
+ */
+ if (n_params > 0)
+ {
+ ErrorContextCallback errcallback;
+
+ params = (ParamListInfo) palloc0(offsetof(ParamListInfoData, params) +
+ n_params * sizeof(ParamExternData));
+ params->numParams = n_params;
+
+ /*
+ * Register error callback to precisely report error in case of conversion error while storig parameter value.
+ */
+ errcallback.callback = prepare_error_callback;
+ errcallback.arg = (void *) psrc;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ for (paramno = 0, binding = binding_list;
+ paramno < n_params;
+ paramno++, binding = binding->next)
+ {
+ Oid ptype = psrc->param_types[paramno];
+
+ param_location = binding->literal->location;
+
+ params->params[paramno].isnull = false;
+ params->params[paramno].value = get_param_value(ptype, &binding->literal->val);
+ /*
+ * We mark the params as CONST. This ensures that any custom plan
+ * makes full use of the parameter values.
+ */
+ params->params[paramno].pflags = PARAM_FLAG_CONST;
+ params->params[paramno].ptype = ptype;
+ }
+ error_context_stack = errcallback.previous;
+ }
+ else
+ {
+ params = NULL;
+ }
+
+ /* Done storing stuff in portal's context */
+ MemoryContextSwitchTo(old_context);
+
+ /*
+ * Obtain a plan from the CachedPlanSource. Any cruft from (re)planning
+ * will be generated in MessageContext. The plan refcount will be
+ * assigned to the Portal, so it will be released at portal destruction.
+ */
+ cplan = GetCachedPlan(psrc, params, false, NULL);
+
+ /*
+ * Now we can define the portal.
+ *
+ * DO NOT put any code that could possibly throw an error between the
+ * above GetCachedPlan call and here.
+ */
+ PortalDefineQuery(portal,
+ NULL,
+ query_string,
+ psrc->commandTag,
+ cplan->stmt_list,
+ cplan);
+
+ /* Done with the snapshot used for parameter I/O and parsing/planning */
+ if (snapshot_set)
+ {
+ PopActiveSnapshot();
+ }
+
+ /*
+ * And we're ready to start portal execution.
+ */
+ PortalStart(portal, params, 0, InvalidSnapshot);
+
+ /*
+ * Apply the result format requests to the portal.
+ */
+ PortalSetResultFormat(portal, 1, &entry->format);
+
+ /*
+ * Finally execute prepared statement
+ */
+ exec_prepared_plan(portal, "", FETCH_ALL, whereToSendOutput);
+
+ /*
+ * Close down transaction statement, if one is open.
+ */
+ finish_xact_command();
+
+ autoprepare_hits += 1;
+
+ return true;
+}
+
+
+void ResetAutoprepareCache(void)
+{
+ if (plan_cache_hash != NULL)
+ {
+ hash_destroy(plan_cache_hash);
+ MemoryContextReset(plan_cache_context);
+ dlist_init(&plan_cache_lru);
+ autoprepare_cached_plans = 0;
+ plan_cache_hash = 0;
+ }
+}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 775477c..94ee570 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -684,10 +684,12 @@ standard_ProcessUtility(PlannedStmt *pstmt,
case T_AlterSystemStmt:
PreventTransactionChain(isTopLevel, "ALTER SYSTEM");
AlterSystemSetConfigFile((AlterSystemStmt *) parsetree);
+ ResetAutoprepareCache();
break;
case T_VariableSetStmt:
ExecSetVariableStmt((VariableSetStmt *) parsetree, isTopLevel);
+ ResetAutoprepareCache();
break;
case T_VariableShowStmt:
@@ -960,6 +962,8 @@ ProcessUtilitySlow(ParseState *pstate,
/* All event trigger calls are done only when isCompleteQuery is true */
needCleanup = isCompleteQuery && EventTriggerBeginCompleteQuery();
+ ResetAutoprepareCache();
+
/* PG_TRY block is to ensure we call EventTriggerEndCompleteQuery */
PG_TRY();
{
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index d0e54b8..af641f0 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -111,6 +111,7 @@
#include "utils/relmapper.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+#include "tcop/pquery.h"
/*
@@ -644,6 +645,7 @@ InvalidateSystemCaches(void)
InvalidateCatalogSnapshot();
ResetCatalogCaches();
+ ResetAutoprepareCache();
RelationCacheInvalidate(); /* gets smgr and relmap too */
for (i = 0; i < syscache_callback_count; i++)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 246fea8..a991c6f 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -472,6 +472,10 @@ int tcp_keepalives_idle;
int tcp_keepalives_interval;
int tcp_keepalives_count;
+
+int autoprepare_threshold;
+int autoprepare_limit;
+
/*
* SSL renegotiation was been removed in PostgreSQL 9.5, but we tolerate it
* being set to zero (meaning never renegotiate) for backward compatibility.
@@ -1958,6 +1962,28 @@ static struct config_int ConfigureNamesInt[] =
check_max_stack_depth, assign_max_stack_depth, NULL
},
+ /*
+ * Threshold for implicit preparing of frequently executed queries
+ */
+ {
+ {"autoprepare_threshold", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Threshold for autopreparing query."),
+ gettext_noop("0 value disables autoprepare.")
+ },
+ &autoprepare_threshold,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+ {
+ {"autoprepare_limit", PGC_USERSET, QUERY_TUNING_OTHER,
+ gettext_noop("Maximal number of autoprepared queries."),
+ gettext_noop("0 means unlimited number of autoprepared queries. Too large number of prepared queries can cause backend memory overflow and slowdown execution speed (because of increased lookup time)")
+ },
+ &autoprepare_limit,
+ 113, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"temp_file_limit", PGC_SUSET, RESOURCES_DISK,
gettext_noop("Limits the total size of all temporary files used by each process."),
diff --git a/src/include/nodes/nodeFuncs.h b/src/include/nodes/nodeFuncs.h
index 3366983..263584d 100644
--- a/src/include/nodes/nodeFuncs.h
+++ b/src/include/nodes/nodeFuncs.h
@@ -73,6 +73,9 @@ extern Node *query_or_expression_tree_mutator(Node *node, Node *(*mutator) (),
extern bool raw_expression_tree_walker(Node *node, bool (*walker) (),
void *context);
+extern bool raw_expression_tree_mutator(Node *node, bool (*mutator) (),
+ void *context);
+
struct PlanState;
extern bool planstate_tree_walker(struct PlanState *planstate, bool (*walker) (),
void *context);
diff --git a/src/include/tcop/pquery.h b/src/include/tcop/pquery.h
index 6abfe7b..d25e744 100644
--- a/src/include/tcop/pquery.h
+++ b/src/include/tcop/pquery.h
@@ -42,4 +42,6 @@ extern uint64 PortalRunFetch(Portal portal,
long count,
DestReceiver *dest);
+extern void ResetAutoprepareCache(void);
+
#endif /* PQUERY_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index c1870d2..cd6a645 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -251,6 +251,9 @@ extern int client_min_messages;
extern int log_min_duration_statement;
extern int log_temp_files;
+extern int autoprepare_threshold;
+extern int autoprepare_limit;
+
extern int temp_file_limit;
extern int num_temp_buffers;
diff --git a/src/test/regress/expected/date_1.out b/src/test/regress/expected/date_1.out
new file mode 100644
index 0000000..b6101c7
--- /dev/null
+++ b/src/test/regress/expected/date_1.out
@@ -0,0 +1,1477 @@
+--
+-- DATE
+--
+CREATE TABLE DATE_TBL (f1 date);
+INSERT INTO DATE_TBL VALUES ('1957-04-09');
+INSERT INTO DATE_TBL VALUES ('1957-06-13');
+INSERT INTO DATE_TBL VALUES ('1996-02-28');
+INSERT INTO DATE_TBL VALUES ('1996-02-29');
+INSERT INTO DATE_TBL VALUES ('1996-03-01');
+INSERT INTO DATE_TBL VALUES ('1996-03-02');
+INSERT INTO DATE_TBL VALUES ('1997-02-28');
+INSERT INTO DATE_TBL VALUES ('1997-02-29');
+ERROR: date/time field value out of range: "1997-02-29"
+LINE 1: INSERT INTO DATE_TBL VALUES ('1997-02-29');
+ ^
+INSERT INTO DATE_TBL VALUES ('1997-03-01');
+INSERT INTO DATE_TBL VALUES ('1997-03-02');
+INSERT INTO DATE_TBL VALUES ('2000-04-01');
+INSERT INTO DATE_TBL VALUES ('2000-04-02');
+INSERT INTO DATE_TBL VALUES ('2000-04-03');
+INSERT INTO DATE_TBL VALUES ('2038-04-08');
+INSERT INTO DATE_TBL VALUES ('2039-04-09');
+INSERT INTO DATE_TBL VALUES ('2040-04-10');
+SELECT f1 AS "Fifteen" FROM DATE_TBL;
+ Fifteen
+------------
+ 04-09-1957
+ 06-13-1957
+ 02-28-1996
+ 02-29-1996
+ 03-01-1996
+ 03-02-1996
+ 02-28-1997
+ 03-01-1997
+ 03-02-1997
+ 04-01-2000
+ 04-02-2000
+ 04-03-2000
+ 04-08-2038
+ 04-09-2039
+ 04-10-2040
+(15 rows)
+
+SELECT f1 AS "Nine" FROM DATE_TBL WHERE f1 < '2000-01-01';
+ Nine
+------------
+ 04-09-1957
+ 06-13-1957
+ 02-28-1996
+ 02-29-1996
+ 03-01-1996
+ 03-02-1996
+ 02-28-1997
+ 03-01-1997
+ 03-02-1997
+(9 rows)
+
+SELECT f1 AS "Three" FROM DATE_TBL
+ WHERE f1 BETWEEN '2000-01-01' AND '2001-01-01';
+ Three
+------------
+ 04-01-2000
+ 04-02-2000
+ 04-03-2000
+(3 rows)
+
+--
+-- Check all the documented input formats
+--
+SET datestyle TO iso; -- display results in ISO
+SET datestyle TO ymd;
+SELECT date 'January 8, 1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999-01-08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999-01-18';
+ date
+------------
+ 1999-01-18
+(1 row)
+
+SELECT date '1/8/1999';
+ERROR: date/time field value out of range: "1/8/1999"
+LINE 1: SELECT date '1/8/1999';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '1/18/1999';
+ERROR: date/time field value out of range: "1/18/1999"
+LINE 1: SELECT date '1/18/1999';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '18/1/1999';
+ERROR: date/time field value out of range: "18/1/1999"
+LINE 1: SELECT date '18/1/1999';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '01/02/03';
+ date
+------------
+ 2001-02-03
+(1 row)
+
+SELECT date '19990108';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '990108';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999.008';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'J2451187';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'January 8, 99 BC';
+ERROR: date/time field value out of range: "January 8, 99 BC"
+LINE 1: SELECT date 'January 8, 99 BC';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '99-Jan-08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999-Jan-08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08-Jan-99';
+ERROR: date/time field value out of range: "08-Jan-99"
+LINE 1: SELECT date '08-Jan-99';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '08-Jan-1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'Jan-08-99';
+ERROR: date/time field value out of range: "Jan-08-99"
+LINE 1: SELECT date 'Jan-08-99';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date 'Jan-08-1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '99-08-Jan';
+ERROR: invalid input syntax for type date: "99-08-Jan"
+LINE 1: SELECT date '99-08-Jan';
+ ^
+SELECT date '1999-08-Jan';
+ERROR: invalid input syntax for type date: "1999-08-Jan"
+LINE 1: SELECT date '1999-08-Jan';
+ ^
+SELECT date '99 Jan 08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999 Jan 08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08 Jan 99';
+ERROR: date/time field value out of range: "08 Jan 99"
+LINE 1: SELECT date '08 Jan 99';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '08 Jan 1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'Jan 08 99';
+ERROR: date/time field value out of range: "Jan 08 99"
+LINE 1: SELECT date 'Jan 08 99';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date 'Jan 08 1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '99 08 Jan';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999 08 Jan';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '99-01-08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999-01-08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08-01-99';
+ERROR: date/time field value out of range: "08-01-99"
+LINE 1: SELECT date '08-01-99';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '08-01-1999';
+ERROR: date/time field value out of range: "08-01-1999"
+LINE 1: SELECT date '08-01-1999';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '01-08-99';
+ERROR: date/time field value out of range: "01-08-99"
+LINE 1: SELECT date '01-08-99';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '01-08-1999';
+ERROR: date/time field value out of range: "01-08-1999"
+LINE 1: SELECT date '01-08-1999';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '99-08-01';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '1999-08-01';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '99 01 08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999 01 08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08 01 99';
+ERROR: date/time field value out of range: "08 01 99"
+LINE 1: SELECT date '08 01 99';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '08 01 1999';
+ERROR: date/time field value out of range: "08 01 1999"
+LINE 1: SELECT date '08 01 1999';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '01 08 99';
+ERROR: date/time field value out of range: "01 08 99"
+LINE 1: SELECT date '01 08 99';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '01 08 1999';
+ERROR: date/time field value out of range: "01 08 1999"
+LINE 1: SELECT date '01 08 1999';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '99 08 01';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '1999 08 01';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SET datestyle TO dmy;
+SELECT date 'January 8, 1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999-01-08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999-01-18';
+ date
+------------
+ 1999-01-18
+(1 row)
+
+SELECT date '1/8/1999';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '1/18/1999';
+ERROR: date/time field value out of range: "1/18/1999"
+LINE 1: SELECT date '1/18/1999';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '18/1/1999';
+ date
+------------
+ 1999-01-18
+(1 row)
+
+SELECT date '01/02/03';
+ date
+------------
+ 2003-02-01
+(1 row)
+
+SELECT date '19990108';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '990108';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999.008';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'J2451187';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'January 8, 99 BC';
+ date
+---------------
+ 0099-01-08 BC
+(1 row)
+
+SELECT date '99-Jan-08';
+ERROR: date/time field value out of range: "99-Jan-08"
+LINE 1: SELECT date '99-Jan-08';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '1999-Jan-08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08-Jan-99';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08-Jan-1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'Jan-08-99';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'Jan-08-1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '99-08-Jan';
+ERROR: invalid input syntax for type date: "99-08-Jan"
+LINE 1: SELECT date '99-08-Jan';
+ ^
+SELECT date '1999-08-Jan';
+ERROR: invalid input syntax for type date: "1999-08-Jan"
+LINE 1: SELECT date '1999-08-Jan';
+ ^
+SELECT date '99 Jan 08';
+ERROR: date/time field value out of range: "99 Jan 08"
+LINE 1: SELECT date '99 Jan 08';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '1999 Jan 08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08 Jan 99';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08 Jan 1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'Jan 08 99';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'Jan 08 1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '99 08 Jan';
+ERROR: invalid input syntax for type date: "99 08 Jan"
+LINE 1: SELECT date '99 08 Jan';
+ ^
+SELECT date '1999 08 Jan';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '99-01-08';
+ERROR: date/time field value out of range: "99-01-08"
+LINE 1: SELECT date '99-01-08';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '1999-01-08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08-01-99';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08-01-1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '01-08-99';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '01-08-1999';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '99-08-01';
+ERROR: date/time field value out of range: "99-08-01"
+LINE 1: SELECT date '99-08-01';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '1999-08-01';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '99 01 08';
+ERROR: date/time field value out of range: "99 01 08"
+LINE 1: SELECT date '99 01 08';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '1999 01 08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08 01 99';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08 01 1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '01 08 99';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '01 08 1999';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '99 08 01';
+ERROR: date/time field value out of range: "99 08 01"
+LINE 1: SELECT date '99 08 01';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '1999 08 01';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SET datestyle TO mdy;
+SELECT date 'January 8, 1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999-01-08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999-01-18';
+ date
+------------
+ 1999-01-18
+(1 row)
+
+SELECT date '1/8/1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1/18/1999';
+ date
+------------
+ 1999-01-18
+(1 row)
+
+SELECT date '18/1/1999';
+ERROR: date/time field value out of range: "18/1/1999"
+LINE 1: SELECT date '18/1/1999';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '01/02/03';
+ date
+------------
+ 2003-01-02
+(1 row)
+
+SELECT date '19990108';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '990108';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '1999.008';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'J2451187';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'January 8, 99 BC';
+ date
+---------------
+ 0099-01-08 BC
+(1 row)
+
+SELECT date '99-Jan-08';
+ERROR: date/time field value out of range: "99-Jan-08"
+LINE 1: SELECT date '99-Jan-08';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '1999-Jan-08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08-Jan-99';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08-Jan-1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'Jan-08-99';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'Jan-08-1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '99-08-Jan';
+ERROR: invalid input syntax for type date: "99-08-Jan"
+LINE 1: SELECT date '99-08-Jan';
+ ^
+SELECT date '1999-08-Jan';
+ERROR: invalid input syntax for type date: "1999-08-Jan"
+LINE 1: SELECT date '1999-08-Jan';
+ ^
+SELECT date '99 Jan 08';
+ERROR: invalid input syntax for type date: "99 Jan 08"
+LINE 1: SELECT date '99 Jan 08';
+ ^
+SELECT date '1999 Jan 08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08 Jan 99';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08 Jan 1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'Jan 08 99';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date 'Jan 08 1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '99 08 Jan';
+ERROR: invalid input syntax for type date: "99 08 Jan"
+LINE 1: SELECT date '99 08 Jan';
+ ^
+SELECT date '1999 08 Jan';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '99-01-08';
+ERROR: date/time field value out of range: "99-01-08"
+LINE 1: SELECT date '99-01-08';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '1999-01-08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08-01-99';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '08-01-1999';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '01-08-99';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '01-08-1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '99-08-01';
+ERROR: date/time field value out of range: "99-08-01"
+LINE 1: SELECT date '99-08-01';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '1999-08-01';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '99 01 08';
+ERROR: date/time field value out of range: "99 01 08"
+LINE 1: SELECT date '99 01 08';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '1999 01 08';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '08 01 99';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '08 01 1999';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+SELECT date '01 08 99';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '01 08 1999';
+ date
+------------
+ 1999-01-08
+(1 row)
+
+SELECT date '99 08 01';
+ERROR: date/time field value out of range: "99 08 01"
+LINE 1: SELECT date '99 08 01';
+ ^
+HINT: Perhaps you need a different "datestyle" setting.
+SELECT date '1999 08 01';
+ date
+------------
+ 1999-08-01
+(1 row)
+
+-- Check upper and lower limits of date range
+SELECT date '4714-11-24 BC';
+ date
+---------------
+ 4714-11-24 BC
+(1 row)
+
+SELECT date '4714-11-23 BC'; -- out of range
+ERROR: date out of range: "4714-11-23 BC"
+LINE 1: SELECT date '4714-11-23 BC';
+ ^
+SELECT date '5874897-12-31';
+ date
+---------------
+ 5874897-12-31
+(1 row)
+
+SELECT date '5874898-01-01'; -- out of range
+ERROR: date out of range: "5874898-01-01"
+LINE 1: SELECT date '5874898-01-01';
+ ^
+RESET datestyle;
+--
+-- Simple math
+-- Leave most of it for the horology tests
+--
+SELECT f1 - date '2000-01-01' AS "Days From 2K" FROM DATE_TBL;
+ Days From 2K
+--------------
+ -15607
+ -15542
+ -1403
+ -1402
+ -1401
+ -1400
+ -1037
+ -1036
+ -1035
+ 91
+ 92
+ 93
+ 13977
+ 14343
+ 14710
+(15 rows)
+
+SELECT f1 - date 'epoch' AS "Days From Epoch" FROM DATE_TBL;
+ Days From Epoch
+-----------------
+ -4650
+ -4585
+ 9554
+ 9555
+ 9556
+ 9557
+ 9920
+ 9921
+ 9922
+ 11048
+ 11049
+ 11050
+ 24934
+ 25300
+ 25667
+(15 rows)
+
+SELECT date 'yesterday' - date 'today' AS "One day";
+ One day
+---------
+ -1
+(1 row)
+
+SELECT date 'today' - date 'tomorrow' AS "One day";
+ One day
+---------
+ -1
+(1 row)
+
+SELECT date 'yesterday' - date 'tomorrow' AS "Two days";
+ Two days
+----------
+ -2
+(1 row)
+
+SELECT date 'tomorrow' - date 'today' AS "One day";
+ One day
+---------
+ 1
+(1 row)
+
+SELECT date 'today' - date 'yesterday' AS "One day";
+ One day
+---------
+ 1
+(1 row)
+
+SELECT date 'tomorrow' - date 'yesterday' AS "Two days";
+ Two days
+----------
+ 2
+(1 row)
+
+--
+-- test extract!
+--
+-- epoch
+--
+SELECT EXTRACT(EPOCH FROM DATE '1970-01-01'); -- 0
+ date_part
+-----------
+ 0
+(1 row)
+
+SELECT EXTRACT(EPOCH FROM TIMESTAMP '1970-01-01'); -- 0
+ date_part
+-----------
+ 0
+(1 row)
+
+SELECT EXTRACT(EPOCH FROM TIMESTAMPTZ '1970-01-01+00'); -- 0
+ date_part
+-----------
+ 0
+(1 row)
+
+--
+-- century
+--
+SELECT EXTRACT(CENTURY FROM DATE '0101-12-31 BC'); -- -2
+ date_part
+-----------
+ -2
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM DATE '0100-12-31 BC'); -- -1
+ date_part
+-----------
+ -1
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM DATE '0001-12-31 BC'); -- -1
+ date_part
+-----------
+ -1
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM DATE '0001-01-01'); -- 1
+ date_part
+-----------
+ 1
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM DATE '0001-01-01 AD'); -- 1
+ date_part
+-----------
+ 1
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM DATE '1900-12-31'); -- 19
+ date_part
+-----------
+ 19
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM DATE '1901-01-01'); -- 20
+ date_part
+-----------
+ 20
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM DATE '2000-12-31'); -- 20
+ date_part
+-----------
+ 20
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM DATE '2001-01-01'); -- 21
+ date_part
+-----------
+ 21
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM CURRENT_DATE)>=21 AS True; -- true
+ true
+------
+ t
+(1 row)
+
+--
+-- millennium
+--
+SELECT EXTRACT(MILLENNIUM FROM DATE '0001-12-31 BC'); -- -1
+ date_part
+-----------
+ -1
+(1 row)
+
+SELECT EXTRACT(MILLENNIUM FROM DATE '0001-01-01 AD'); -- 1
+ date_part
+-----------
+ 1
+(1 row)
+
+SELECT EXTRACT(MILLENNIUM FROM DATE '1000-12-31'); -- 1
+ date_part
+-----------
+ 1
+(1 row)
+
+SELECT EXTRACT(MILLENNIUM FROM DATE '1001-01-01'); -- 2
+ date_part
+-----------
+ 2
+(1 row)
+
+SELECT EXTRACT(MILLENNIUM FROM DATE '2000-12-31'); -- 2
+ date_part
+-----------
+ 2
+(1 row)
+
+SELECT EXTRACT(MILLENNIUM FROM DATE '2001-01-01'); -- 3
+ date_part
+-----------
+ 3
+(1 row)
+
+-- next test to be fixed on the turn of the next millennium;-)
+SELECT EXTRACT(MILLENNIUM FROM CURRENT_DATE); -- 3
+ date_part
+-----------
+ 3
+(1 row)
+
+--
+-- decade
+--
+SELECT EXTRACT(DECADE FROM DATE '1994-12-25'); -- 199
+ date_part
+-----------
+ 199
+(1 row)
+
+SELECT EXTRACT(DECADE FROM DATE '0010-01-01'); -- 1
+ date_part
+-----------
+ 1
+(1 row)
+
+SELECT EXTRACT(DECADE FROM DATE '0009-12-31'); -- 0
+ date_part
+-----------
+ 0
+(1 row)
+
+SELECT EXTRACT(DECADE FROM DATE '0001-01-01 BC'); -- 0
+ date_part
+-----------
+ 0
+(1 row)
+
+SELECT EXTRACT(DECADE FROM DATE '0002-12-31 BC'); -- -1
+ date_part
+-----------
+ -1
+(1 row)
+
+SELECT EXTRACT(DECADE FROM DATE '0011-01-01 BC'); -- -1
+ date_part
+-----------
+ -1
+(1 row)
+
+SELECT EXTRACT(DECADE FROM DATE '0012-12-31 BC'); -- -2
+ date_part
+-----------
+ -2
+(1 row)
+
+--
+-- some other types:
+--
+-- on a timestamp.
+SELECT EXTRACT(CENTURY FROM NOW())>=21 AS True; -- true
+ true
+------
+ t
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM TIMESTAMP '1970-03-20 04:30:00.00000'); -- 20
+ date_part
+-----------
+ 20
+(1 row)
+
+-- on an interval
+SELECT EXTRACT(CENTURY FROM INTERVAL '100 y'); -- 1
+ date_part
+-----------
+ 1
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM INTERVAL '99 y'); -- 0
+ date_part
+-----------
+ 0
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM INTERVAL '-99 y'); -- 0
+ date_part
+-----------
+ 0
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM INTERVAL '-100 y'); -- -1
+ date_part
+-----------
+ -1
+(1 row)
+
+--
+-- test trunc function!
+--
+SELECT DATE_TRUNC('MILLENNIUM', TIMESTAMP '1970-03-20 04:30:00.00000'); -- 1001
+ date_trunc
+--------------------------
+ Thu Jan 01 00:00:00 1001
+(1 row)
+
+SELECT DATE_TRUNC('MILLENNIUM', DATE '1970-03-20'); -- 1001-01-01
+ date_trunc
+------------------------------
+ Thu Jan 01 00:00:00 1001 PST
+(1 row)
+
+SELECT DATE_TRUNC('CENTURY', TIMESTAMP '1970-03-20 04:30:00.00000'); -- 1901
+ date_trunc
+--------------------------
+ Tue Jan 01 00:00:00 1901
+(1 row)
+
+SELECT DATE_TRUNC('CENTURY', DATE '1970-03-20'); -- 1901
+ date_trunc
+------------------------------
+ Tue Jan 01 00:00:00 1901 PST
+(1 row)
+
+SELECT DATE_TRUNC('CENTURY', DATE '2004-08-10'); -- 2001-01-01
+ date_trunc
+------------------------------
+ Mon Jan 01 00:00:00 2001 PST
+(1 row)
+
+SELECT DATE_TRUNC('CENTURY', DATE '0002-02-04'); -- 0001-01-01
+ date_trunc
+------------------------------
+ Mon Jan 01 00:00:00 0001 PST
+(1 row)
+
+SELECT DATE_TRUNC('CENTURY', DATE '0055-08-10 BC'); -- 0100-01-01 BC
+ date_trunc
+---------------------------------
+ Tue Jan 01 00:00:00 0100 PST BC
+(1 row)
+
+SELECT DATE_TRUNC('DECADE', DATE '1993-12-25'); -- 1990-01-01
+ date_trunc
+------------------------------
+ Mon Jan 01 00:00:00 1990 PST
+(1 row)
+
+SELECT DATE_TRUNC('DECADE', DATE '0004-12-25'); -- 0001-01-01 BC
+ date_trunc
+---------------------------------
+ Sat Jan 01 00:00:00 0001 PST BC
+(1 row)
+
+SELECT DATE_TRUNC('DECADE', DATE '0002-12-31 BC'); -- 0011-01-01 BC
+ date_trunc
+---------------------------------
+ Mon Jan 01 00:00:00 0011 PST BC
+(1 row)
+
+--
+-- test infinity
+--
+select 'infinity'::date, '-infinity'::date;
+ date | date
+----------+-----------
+ infinity | -infinity
+(1 row)
+
+select 'infinity'::date > 'today'::date as t;
+ t
+---
+ t
+(1 row)
+
+select '-infinity'::date < 'today'::date as t;
+ t
+---
+ t
+(1 row)
+
+select isfinite('infinity'::date), isfinite('-infinity'::date), isfinite('today'::date);
+ isfinite | isfinite | isfinite
+----------+----------+----------
+ f | f | t
+(1 row)
+
+--
+-- oscillating fields from non-finite date/timestamptz:
+--
+SELECT EXTRACT(HOUR FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(HOUR FROM DATE '-infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(HOUR FROM TIMESTAMP 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(HOUR FROM TIMESTAMP '-infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(HOUR FROM TIMESTAMPTZ 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(HOUR FROM TIMESTAMPTZ '-infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+-- all possible fields
+SELECT EXTRACT(MICROSECONDS FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(MILLISECONDS FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(SECOND FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(MINUTE FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(HOUR FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(DAY FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(MONTH FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(QUARTER FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(WEEK FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(DOW FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(ISODOW FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(DOY FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(TIMEZONE FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(TIMEZONE_M FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+SELECT EXTRACT(TIMEZONE_H FROM DATE 'infinity'); -- NULL
+ date_part
+-----------
+
+(1 row)
+
+--
+-- monotonic fields from non-finite date/timestamptz:
+--
+SELECT EXTRACT(EPOCH FROM DATE 'infinity'); -- Infinity
+ date_part
+-----------
+ Infinity
+(1 row)
+
+SELECT EXTRACT(EPOCH FROM DATE '-infinity'); -- -Infinity
+ date_part
+-----------
+ -Infinity
+(1 row)
+
+SELECT EXTRACT(EPOCH FROM TIMESTAMP 'infinity'); -- Infinity
+ date_part
+-----------
+ Infinity
+(1 row)
+
+SELECT EXTRACT(EPOCH FROM TIMESTAMP '-infinity'); -- -Infinity
+ date_part
+-----------
+ -Infinity
+(1 row)
+
+SELECT EXTRACT(EPOCH FROM TIMESTAMPTZ 'infinity'); -- Infinity
+ date_part
+-----------
+ Infinity
+(1 row)
+
+SELECT EXTRACT(EPOCH FROM TIMESTAMPTZ '-infinity'); -- -Infinity
+ date_part
+-----------
+ -Infinity
+(1 row)
+
+-- all possible fields
+SELECT EXTRACT(YEAR FROM DATE 'infinity'); -- Infinity
+ date_part
+-----------
+ Infinity
+(1 row)
+
+SELECT EXTRACT(DECADE FROM DATE 'infinity'); -- Infinity
+ date_part
+-----------
+ Infinity
+(1 row)
+
+SELECT EXTRACT(CENTURY FROM DATE 'infinity'); -- Infinity
+ date_part
+-----------
+ Infinity
+(1 row)
+
+SELECT EXTRACT(MILLENNIUM FROM DATE 'infinity'); -- Infinity
+ date_part
+-----------
+ Infinity
+(1 row)
+
+SELECT EXTRACT(JULIAN FROM DATE 'infinity'); -- Infinity
+ date_part
+-----------
+ Infinity
+(1 row)
+
+SELECT EXTRACT(ISOYEAR FROM DATE 'infinity'); -- Infinity
+ date_part
+-----------
+ Infinity
+(1 row)
+
+SELECT EXTRACT(EPOCH FROM DATE 'infinity'); -- Infinity
+ date_part
+-----------
+ Infinity
+(1 row)
+
+--
+-- wrong fields from non-finite date:
+--
+SELECT EXTRACT(MICROSEC FROM DATE 'infinity'); -- ERROR: timestamp units "microsec" not recognized
+ERROR: timestamp units "microsec" not recognized
+SELECT EXTRACT(UNDEFINED FROM DATE 'infinity'); -- ERROR: timestamp units "undefined" not supported
+ERROR: timestamp units "undefined" not supported
+-- test constructors
+select make_date(2013, 7, 15);
+ make_date
+------------
+ 07-15-2013
+(1 row)
+
+select make_date(-44, 3, 15);
+ make_date
+---------------
+ 03-15-0044 BC
+(1 row)
+
+select make_time(8, 20, 0.0);
+ make_time
+-----------
+ 08:20:00
+(1 row)
+
+-- should fail
+select make_date(2013, 2, 30);
+ERROR: date field value out of range: 2013-02-30
+select make_date(2013, 13, 1);
+ERROR: date field value out of range: 2013-13-01
+select make_date(2013, 11, -1);
+ERROR: date field value out of range: 2013-11--1
+select make_time(10, 55, 100.1);
+ERROR: time field value out of range: 10:55:100.1
+select make_time(24, 0, 2.1);
+ERROR: time field value out of range: 24:00:2.1
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks
@ 2021-01-13 11:53 Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Bharath Rupireddy @ 2021-01-13 11:53 UTC (permalink / raw)
Currently, in logical replication, relation map cache in the
subscriber is not getting invalidated when anything changes in
pg_subscription_rel or pg_subscription catalogues. So we end up
not reading the latest system catalogues always in logicalrep_rel_open
using GetSubscriptionRelState.
To fix this, invalidate the relation map cache entries in
invalidate_syncing_table_states and subscription_change_cb which
are invalidation callbacks for pg_subscription_rel and pg_subscription
catalogues respectively.
---
src/backend/replication/logical/relation.c | 26 +++++++++++++++++++++
src/backend/replication/logical/tablesync.c | 3 +++
src/backend/replication/logical/worker.c | 3 +++
src/include/replication/logicalrelation.h | 1 +
4 files changed, 33 insertions(+)
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index e861c0ff80..88ac772444 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -96,6 +96,32 @@ logicalrep_relmap_invalidate_cb(Datum arg, Oid reloid)
}
}
+/*
+ * Invalidate relation map cache whenever syscache of pg_subscription_rel or
+ * pg_subscription gets changed.
+ */
+void
+logicalrep_relmap_invalidate(void)
+{
+ LogicalRepRelMapEntry *entry;
+ HASH_SEQ_STATUS status;
+
+ /* Just to be sure. */
+ if (LogicalRepRelMap == NULL)
+ return;
+
+ /*
+ * There is no way to find the cache entry for which the syscache has been
+ * changed, so we mark all the cache entries state as unknown. Because of
+ * this, in logicalrep_rel_open the cache entry state will be read from
+ * the system catalogue using GetSubscriptionRelState.
+ */
+ hash_seq_init(&status, LogicalRepRelMap);
+
+ while ((entry = (LogicalRepRelMapEntry *) hash_seq_search(&status)) != NULL)
+ entry->state = SUBREL_STATE_UNKNOWN;
+}
+
/*
* Initialize the relation map cache.
*/
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..6b29173a5c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -257,6 +257,9 @@ void
invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
{
table_states_valid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f2b2549a51..347264e4a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2487,6 +2487,9 @@ static void
subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
{
MySubscriptionValid = false;
+
+ /* Invalidate relation map cache. */
+ logicalrep_relmap_invalidate();
}
/*
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..f5e70b75b0 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,5 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
extern char *logicalrep_typmap_gettypname(Oid remoteid);
+extern void logicalrep_relmap_invalidate(void);
#endif /* LOGICALRELATION_H */
--
2.30.0
--=-=-=--
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Thread-safe nl_langinfo() and localeconv()
@ 2024-08-15 08:03 Thomas Munro <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Thomas Munro @ 2024-08-15 08:03 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: pgsql-hackers
Here's a new patch to add to this pile, this time for check_locale().
I also improved the error handling and comments in the other patches.
Attachments:
[text/x-patch] v3-0001-Provide-thread-safe-pg_localeconv_r.patch (22.5K, ../../CA+hUKG+eEQ-+g2Ht-mho1o2xGX7v+GsuB4OQSUZv2U9S8huW8w@mail.gmail.com/2-v3-0001-Provide-thread-safe-pg_localeconv_r.patch)
download | inline diff:
From 4831ff4373b9d713c78e303d9758de347aadfc2f Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Tue, 13 Aug 2024 14:15:54 +1200
Subject: [PATCH v3 1/3] Provide thread-safe pg_localeconv_r().
This involves four different implementation strategies:
1. For Windows, we now require _configthreadlocale() to be available
and work, and the documentation says that the object returned by
localeconv() is in thread-local memory.
2. For glibc, we translate to nl_langinfo_l() calls, because it offers
the same information that way as an extension, and that API is
thread-safe.
3. For macOS/*BSD, use localeconv_l(), which is thread-safe.
4. For everything else, use uselocale() to set the locale for the
thread, and use a big ugly lock to defend against the returned object
being concurrently clobbered. In practice this currently means only
Solaris.
The new call is used in pg_locale.c, replacing calls to setlocale() and
localeconv().
This patch adds a hard requirement on Windows' _configthreadlocale().
In the past there were said to be MinGW systems that didn't have it, or
had it but it didn't work. As far as I can tell, CI (optional MinGW
task + mingw cross build warning task) and build farm (fairywren msys)
should be happy with this. Fingers crossed. (There are places that use
configure checks for that in ECPG; other proposed patches would remove
those later.)
Reviewed-by: Heikki Linnakangas <[email protected]>
Discussion: https://postgr.es/m/CA%2BhUKGJqVe0%2BPv9dvC9dSums_PXxGo9SWcxYAMBguWJUGbWz-A%40mail.gmail.com
---
configure | 2 +-
configure.ac | 1 +
meson.build | 1 +
src/backend/utils/adt/pg_locale.c | 128 ++---------
src/include/pg_config.h.in | 3 +
src/include/port.h | 6 +
src/port/Makefile | 1 +
src/port/meson.build | 1 +
src/port/pg_localeconv_r.c | 367 ++++++++++++++++++++++++++++++
9 files changed, 402 insertions(+), 108 deletions(-)
create mode 100644 src/port/pg_localeconv_r.c
diff --git a/configure b/configure
index 2abbeb27944..60dcf1e436e 100755
--- a/configure
+++ b/configure
@@ -15232,7 +15232,7 @@ fi
LIBS_including_readline="$LIBS"
LIBS=`echo "$LIBS" | sed -e 's/-ledit//g' -e 's/-lreadline//g'`
-for ac_func in backtrace_symbols copyfile copy_file_range getifaddrs getpeerucred inet_pton kqueue mbstowcs_l memset_s posix_fallocate ppoll pthread_is_threaded_np setproctitle setproctitle_fast strchrnul strsignal syncfs sync_file_range uselocale wcstombs_l
+for ac_func in backtrace_symbols copyfile copy_file_range getifaddrs getpeerucred inet_pton localeconv_l kqueue mbstowcs_l memset_s posix_fallocate ppoll pthread_is_threaded_np setproctitle setproctitle_fast strchrnul strsignal syncfs sync_file_range uselocale wcstombs_l
do :
as_ac_var=`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh`
ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var"
diff --git a/configure.ac b/configure.ac
index c46ed2c591a..59e51a74629 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1735,6 +1735,7 @@ AC_CHECK_FUNCS(m4_normalize([
getifaddrs
getpeerucred
inet_pton
+ localeconv_l
kqueue
mbstowcs_l
memset_s
diff --git a/meson.build b/meson.build
index cd711c6d018..028a14547aa 100644
--- a/meson.build
+++ b/meson.build
@@ -2675,6 +2675,7 @@ func_checks = [
['inet_aton'],
['inet_pton'],
['kqueue'],
+ ['localeconv_l'],
['mbstowcs_l'],
['memset_s'],
['mkdtemp'],
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index cd3661e7279..dd4ba9e0e89 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -543,12 +543,8 @@ PGLC_localeconv(void)
static struct lconv CurrentLocaleConv;
static bool CurrentLocaleConvAllocated = false;
struct lconv *extlconv;
- struct lconv worklconv;
- char *save_lc_monetary;
- char *save_lc_numeric;
-#ifdef WIN32
- char *save_lc_ctype;
-#endif
+ struct lconv tmp;
+ struct lconv worklconv = {0};
/* Did we do it already? */
if (CurrentLocaleConvValid)
@@ -562,77 +558,21 @@ PGLC_localeconv(void)
}
/*
- * This is tricky because we really don't want to risk throwing error
- * while the locale is set to other than our usual settings. Therefore,
- * the process is: collect the usual settings, set locale to special
- * setting, copy relevant data into worklconv using strdup(), restore
- * normal settings, convert data to desired encoding, and finally stash
- * the collected data in CurrentLocaleConv. This makes it safe if we
- * throw an error during encoding conversion or run out of memory anywhere
- * in the process. All data pointed to by struct lconv members is
- * allocated with strdup, to avoid premature elog(ERROR) and to allow
- * using a single cleanup routine.
+ * Use thread-safe method of obtaining a copy of lconv from the operating
+ * system.
*/
- memset(&worklconv, 0, sizeof(worklconv));
-
- /* Save prevailing values of monetary and numeric locales */
- save_lc_monetary = setlocale(LC_MONETARY, NULL);
- if (!save_lc_monetary)
- elog(ERROR, "setlocale(NULL) failed");
- save_lc_monetary = pstrdup(save_lc_monetary);
-
- save_lc_numeric = setlocale(LC_NUMERIC, NULL);
- if (!save_lc_numeric)
- elog(ERROR, "setlocale(NULL) failed");
- save_lc_numeric = pstrdup(save_lc_numeric);
-
-#ifdef WIN32
-
- /*
- * The POSIX standard explicitly says that it is undefined what happens if
- * LC_MONETARY or LC_NUMERIC imply an encoding (codeset) different from
- * that implied by LC_CTYPE. In practice, all Unix-ish platforms seem to
- * believe that localeconv() should return strings that are encoded in the
- * codeset implied by the LC_MONETARY or LC_NUMERIC locale name. Hence,
- * once we have successfully collected the localeconv() results, we will
- * convert them from that codeset to the desired server encoding.
- *
- * Windows, of course, resolutely does things its own way; on that
- * platform LC_CTYPE has to match LC_MONETARY/LC_NUMERIC to get sane
- * results. Hence, we must temporarily set that category as well.
- */
-
- /* Save prevailing value of ctype locale */
- save_lc_ctype = setlocale(LC_CTYPE, NULL);
- if (!save_lc_ctype)
- elog(ERROR, "setlocale(NULL) failed");
- save_lc_ctype = pstrdup(save_lc_ctype);
-
- /* Here begins the critical section where we must not throw error */
-
- /* use numeric to set the ctype */
- setlocale(LC_CTYPE, locale_numeric);
-#endif
-
- /* Get formatting information for numeric */
- setlocale(LC_NUMERIC, locale_numeric);
- extlconv = localeconv();
-
- /* Must copy data now in case setlocale() overwrites it */
+ if (pg_localeconv_r(locale_monetary,
+ locale_numeric,
+ &tmp) != 0)
+ elog(ERROR,
+ "could not get lconv for LC_MONETARY = \"%s\", LC_NUMERIC = \"%s\": %m",
+ locale_monetary, locale_numeric);
+
+ /* Must copy data now now so we can re-encode it. */
+ extlconv = &tmp;
worklconv.decimal_point = strdup(extlconv->decimal_point);
worklconv.thousands_sep = strdup(extlconv->thousands_sep);
worklconv.grouping = strdup(extlconv->grouping);
-
-#ifdef WIN32
- /* use monetary to set the ctype */
- setlocale(LC_CTYPE, locale_monetary);
-#endif
-
- /* Get formatting information for monetary */
- setlocale(LC_MONETARY, locale_monetary);
- extlconv = localeconv();
-
- /* Must copy data now in case setlocale() overwrites it */
worklconv.int_curr_symbol = strdup(extlconv->int_curr_symbol);
worklconv.currency_symbol = strdup(extlconv->currency_symbol);
worklconv.mon_decimal_point = strdup(extlconv->mon_decimal_point);
@@ -650,45 +590,19 @@ PGLC_localeconv(void)
worklconv.p_sign_posn = extlconv->p_sign_posn;
worklconv.n_sign_posn = extlconv->n_sign_posn;
- /*
- * Restore the prevailing locale settings; failure to do so is fatal.
- * Possibly we could limp along with nondefault LC_MONETARY or LC_NUMERIC,
- * but proceeding with the wrong value of LC_CTYPE would certainly be bad
- * news; and considering that the prevailing LC_MONETARY and LC_NUMERIC
- * are almost certainly "C", there's really no reason that restoring those
- * should fail.
- */
-#ifdef WIN32
- if (!setlocale(LC_CTYPE, save_lc_ctype))
- elog(FATAL, "failed to restore LC_CTYPE to \"%s\"", save_lc_ctype);
-#endif
- if (!setlocale(LC_MONETARY, save_lc_monetary))
- elog(FATAL, "failed to restore LC_MONETARY to \"%s\"", save_lc_monetary);
- if (!setlocale(LC_NUMERIC, save_lc_numeric))
- elog(FATAL, "failed to restore LC_NUMERIC to \"%s\"", save_lc_numeric);
+ /* Free the contents of the object populated by pg_localeconv_r(). */
+ pg_localeconv_free(&tmp);
+
+ /* If any of the preceding strdup calls failed, complain now. */
+ if (!struct_lconv_is_valid(&worklconv))
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory")));
- /*
- * At this point we've done our best to clean up, and can call functions
- * that might possibly throw errors with a clean conscience. But let's
- * make sure we don't leak any already-strdup'd fields in worklconv.
- */
PG_TRY();
{
int encoding;
- /* Release the pstrdup'd locale names */
- pfree(save_lc_monetary);
- pfree(save_lc_numeric);
-#ifdef WIN32
- pfree(save_lc_ctype);
-#endif
-
- /* If any of the preceding strdup calls failed, complain now. */
- if (!struct_lconv_is_valid(&worklconv))
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory")));
-
/*
* Now we must perform encoding conversion from whatever's associated
* with the locales into the database encoding. If we can't identify
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 979925cc2e2..f3db06d155f 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -280,6 +280,9 @@
/* Define to 1 if you have the `zstd' library (-lzstd). */
#undef HAVE_LIBZSTD
+/* Define to 1 if you have the `localeconv_l' function. */
+#undef HAVE_LOCALECONV_L
+
/* Define to 1 if `long int' works and is 64 bits. */
#undef HAVE_LONG_INT_64
diff --git a/src/include/port.h b/src/include/port.h
index c7400052675..ac0cff79fc6 100644
--- a/src/include/port.h
+++ b/src/include/port.h
@@ -465,6 +465,12 @@ extern void *bsearch_arg(const void *key, const void *base0,
int (*compar) (const void *, const void *, void *),
void *arg);
+/* port/pg_localeconv_r.c */
+extern int pg_localeconv_r(const char *lc_monetary,
+ const char *lc_numeric,
+ struct lconv *output);
+extern void pg_localeconv_free(struct lconv *lconv);
+
/* port/chklocale.c */
extern int pg_get_encoding_from_locale(const char *ctype, bool write_message);
diff --git a/src/port/Makefile b/src/port/Makefile
index db7c02117b0..f24d2dbc138 100644
--- a/src/port/Makefile
+++ b/src/port/Makefile
@@ -45,6 +45,7 @@ OBJS = \
noblock.o \
path.o \
pg_bitutils.o \
+ pg_localeconv_r.o \
pg_strong_random.o \
pgcheckdir.o \
pgmkdirp.o \
diff --git a/src/port/meson.build b/src/port/meson.build
index ff54b7b53e9..9d4c4018523 100644
--- a/src/port/meson.build
+++ b/src/port/meson.build
@@ -7,6 +7,7 @@ pgport_sources = [
'noblock.c',
'path.c',
'pg_bitutils.c',
+ 'pg_localeconv_r.c',
'pg_strong_random.c',
'pgcheckdir.c',
'pgmkdirp.c',
diff --git a/src/port/pg_localeconv_r.c b/src/port/pg_localeconv_r.c
new file mode 100644
index 00000000000..efb98cd127d
--- /dev/null
+++ b/src/port/pg_localeconv_r.c
@@ -0,0 +1,367 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_localeconv_r.c
+ * Thread-safe implementations of localeconv()
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/port/pg_localeconv_r.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "c.h"
+
+#if !defined(WIN32)
+#include <langinfo.h>
+#include <pthread.h>
+#endif
+
+#include <limits.h>
+
+#ifdef MON_THOUSANDS_SEP
+/*
+ * One of glibc's extended langinfo items detected. Assume that the full set
+ * is present, which means we can use nl_langinfo_l() instead of localeconv().
+ */
+#define TRANSLATE_FROM_LANGINFO
+#endif
+
+struct lconv_member_info
+{
+ bool is_string;
+ int category;
+ size_t offset;
+#ifdef TRANSLATE_FROM_LANGINFO
+ nl_item item;
+#endif
+};
+
+/* Some macros to declare the lconv members compactly. */
+#ifdef TRANSLATE_FROM_LANGINFO
+#define LCONV_M(is_string, category, name, item) \
+ { is_string, category, offsetof(struct lconv, name), item }
+#else
+#define LCONV_M(is_string, category, name, item) \
+ { is_string, category, offsetof(struct lconv, name) }
+#endif
+#define LCONV_S(c, n, i) LCONV_M(true, c, n, i)
+#define LCONV_C(c, n, i) LCONV_M(false, c, n, i)
+
+/*
+ * The work of populating lconv objects is driven by this table. Since we
+ * tolerate non-matching encodings in LC_NUMERIC and LC_MONETARY, we have to
+ * call the underlying OS routine multiple times, with the correct locales.
+ * The first column of this table says which locale applies to each struct
+ * member. The second column is the name of the struct member. The third
+ * column is the name of the nl_item, if translating from nl_langinfo_l() (it's
+ * always the member name, in upper case).
+ */
+const static struct lconv_member_info table[] = {
+ /* String fields. */
+ LCONV_S(LC_NUMERIC, decimal_point, DECIMAL_POINT),
+ LCONV_S(LC_NUMERIC, thousands_sep, THOUSANDS_SEP),
+ LCONV_S(LC_NUMERIC, grouping, GROUPING),
+ LCONV_S(LC_MONETARY, int_curr_symbol, INT_CURR_SYMBOL),
+ LCONV_S(LC_MONETARY, currency_symbol, CURRENCY_SYMBOL),
+ LCONV_S(LC_MONETARY, mon_decimal_point, MON_DECIMAL_POINT),
+ LCONV_S(LC_MONETARY, mon_thousands_sep, MON_THOUSANDS_SEP),
+ LCONV_S(LC_MONETARY, mon_grouping, MON_GROUPING),
+ LCONV_S(LC_MONETARY, positive_sign, POSITIVE_SIGN),
+ LCONV_S(LC_MONETARY, negative_sign, NEGATIVE_SIGN),
+
+ /* Character fields. */
+ LCONV_C(LC_MONETARY, int_frac_digits, INT_FRAC_DIGITS),
+ LCONV_C(LC_MONETARY, frac_digits, FRAC_DIGITS),
+ LCONV_C(LC_MONETARY, p_cs_precedes, P_CS_PRECEDES),
+ LCONV_C(LC_MONETARY, p_sep_by_space, P_SEP_BY_SPACE),
+ LCONV_C(LC_MONETARY, n_cs_precedes, N_CS_PRECEDES),
+ LCONV_C(LC_MONETARY, n_sep_by_space, N_SEP_BY_SPACE),
+ LCONV_C(LC_MONETARY, p_sign_posn, P_SIGN_POSN),
+ LCONV_C(LC_MONETARY, n_sign_posn, N_SIGN_POSN),
+};
+
+static inline char **
+lconv_string_member(struct lconv *lconv, int i)
+{
+ return (char **) ((char *) lconv + table[i].offset);
+}
+
+static inline char *
+lconv_char_member(struct lconv *lconv, int i)
+{
+ return (char *) lconv + table[i].offset;
+}
+
+/*
+ * Free the members of a struct lconv populated by pg_localeconv_r(). The
+ * struct itself is in storage provided by the caller of pg_localeconv_r().
+ */
+void
+pg_localeconv_free(struct lconv *lconv)
+{
+ for (int i = 0; i < lengthof(table); ++i)
+ if (table[i].is_string)
+ free(*lconv_string_member(lconv, i));
+}
+
+#ifdef TRANSLATE_FROM_LANGINFO
+/*
+ * Fill in struct lconv members using the equivalent nl_langinfo_l() items.
+ */
+static int
+pg_localeconv_from_langinfo(struct lconv *dst,
+ locale_t monetary_locale,
+ locale_t numeric_locale)
+{
+ for (int i = 0; i < lengthof(table); ++i)
+ {
+ locale_t locale;
+
+ locale = table[i].category == LC_NUMERIC ?
+ numeric_locale : monetary_locale;
+
+ if (table[i].is_string)
+ {
+ char *string;
+
+ string = nl_langinfo_l(table[i].item, locale);
+ if (!(string = strdup(string)))
+ {
+ pg_localeconv_free(dst);
+ errno = ENOMEM;
+ return -1;
+ }
+ *lconv_string_member(dst, i) = string;
+ }
+ else
+ {
+ *lconv_char_member(dst, i) =
+ *nl_langinfo_l(table[i].item, locale);
+ }
+ }
+
+ return 0;
+}
+#else
+/*
+ * Copy members from a given category. Note that you have to call this twice
+ * to copy the LC_MONETARY and then LC_NUMERIC members.
+ */
+static int
+pg_localeconv_copy_members(struct lconv *dst,
+ struct lconv *src,
+ int category)
+{
+ for (int i = 0; i < lengthof(table); ++i)
+ {
+ if (table[i].category != category)
+ continue;
+
+ if (table[i].is_string)
+ {
+ char *string;
+
+ string = *lconv_string_member(src, i);
+ if (!(string = strdup(string)))
+ {
+ pg_localeconv_free(dst);
+ errno = ENOMEM;
+ return -1;
+ }
+ *lconv_string_member(dst, i) = string;
+ }
+ else
+ {
+ *lconv_char_member(dst, i) = *lconv_char_member(src, i);
+ }
+ }
+
+ return 0;
+}
+#endif
+
+/*
+ * A thread-safe routine to get a copy of the lconv struct for a given
+ * LC_NUMERIC and LC_MONETARY. Different approaches are used on different
+ * OSes, because the standard interface is so multi-threading unfriendly.
+ *
+ * 1. On Windows, there is no uselocale(), but there is a way to put
+ * setlocale() into a thread-local mode temporarily. Its localeconv() is
+ * documented as returning a pointer to thread-local storage, so we don't have
+ * to worry about concurrent callers.
+ *
+ * 2. On Glibc, as an extension, all the information required to populate
+ * struct lconv is also available via nl_langpath_l(), which is thread-safe.
+ *
+ * 3. On macOS and *BSD, there is localeconv_l(), so we can create a temporary
+ * locale_t to pass in, and the result is a pointer to storage associated with
+ * the locale_t so we control its lifetime and we don't have to worry about
+ * concurrent calls clobbering it.
+ *
+ * 4. Otherwise, we wrap plain old localeconv() in uselocale() to avoid
+ * touching the global locale, but the output buffer is allowed by the standard
+ * to be overwritten by concurrent calls to localeconv(). We protect against
+ * _this_ function doing that with a Big Lock, but there isn't much we can do
+ * about code outside our tree that might call localeconv(), given such a poor
+ * interface.
+ *
+ * The POSIX standard explicitly says that it is undefined what happens if
+ * LC_MONETARY or LC_NUMERIC imply an encoding (codeset) different from that
+ * implied by LC_CTYPE. In practice, all Unix-ish platforms seem to believe
+ * that localeconv() should return strings that are encoded in the codeset
+ * implied by the LC_MONETARY or LC_NUMERIC locale name. On Windows, LC_CTYPE
+ * has to match to get sane results.
+ *
+ * To get predicable results on all platforms, we'll call the underlying
+ * routines with LC_ALL set to the appropriate locale for each set of members,
+ * and merge the results. Three members of the resulting object are therefore
+ * guaranteed to be encoded with LC_NUMERIC's codeset: "decimal_point",
+ * "thousands_sep" and "grouping". All other members are encoded with
+ * LC_MONETARY's codeset.
+ *
+ * Returns 0 on success. Returns non-zero on failure, and sets errno. On
+ * success, the caller is responsible for calling pg_localeconf_free() on the
+ * output struct to free the string members it contains.
+ */
+int
+pg_localeconv_r(const char *lc_monetary,
+ const char *lc_numeric,
+ struct lconv *output)
+{
+#ifdef WIN32
+ wchar_t *save_lc_ctype = NULL;
+ wchar_t *save_lc_monetary = NULL;
+ wchar_t *save_lc_numeric = NULL;
+ int save_config_thread_locale;
+ int result = -1;
+
+ /* Put setlocale() into thread-local mode. */
+ save_config_thread_locale = _configthreadlocale(_ENABLE_PER_THREAD_LOCALE);
+
+ /*
+ * Capture the current values as wide strings. Otherwise, we might not be
+ * able to restore them if their names contain non-ASCII characters and
+ * the intermediate locale changes the expected encoding. We don't want
+ * to leave the caller in an unexpected state by failing to restore, or
+ * crash the runtime library.
+ */
+ save_lc_ctype = _wsetlocale(LC_CTYPE, NULL);
+ if (!save_lc_ctype || !(save_lc_ctype = wcsdup(save_lc_ctype)))
+ goto exit;
+ save_lc_monetary = _wsetlocale(LC_MONETARY, NULL);
+ if (!save_lc_monetary || !(save_lc_monetary = wcsdup(save_lc_monetary)))
+ goto exit;
+ save_lc_numeric = _wsetlocale(LC_NUMERIC, NULL);
+ if (!save_lc_numeric || !(save_lc_numeric = wcsdup(save_lc_numeric)))
+ goto exit;
+
+ memset(output, 0, sizeof(*output));
+
+ /* Copy the LC_MONETARY members. */
+ if (!setlocale(LC_ALL, lc_monetary))
+ goto exit;
+ result = pg_localeconv_copy_members(output, localeconv(), LC_MONETARY);
+ if (result != 0)
+ goto exit;
+
+ /* Copy the LC_NUMERIC members. */
+ if (!setlocale(LC_ALL, lc_numeric))
+ goto exit;
+ result = pg_localeconv_copy_members(output, localeconv(), LC_NUMERIC);
+
+exit:
+ /* Restore everything we changed. */
+ if (save_lc_ctype)
+ {
+ _wsetlocale(LC_CTYPE, save_lc_ctype);
+ free(save_lc_ctype);
+ }
+ if (save_lc_monetary)
+ {
+ _wsetlocale(LC_MONETARY, save_lc_monetary);
+ free(save_lc_monetary);
+ }
+ if (save_lc_numeric)
+ {
+ _wsetlocale(LC_NUMERIC, save_lc_numeric);
+ free(save_lc_numeric);
+ }
+ _configthreadlocale(save_config_thread_locale);
+
+ return result;
+
+#else
+ locale_t monetary_locale;
+ locale_t numeric_locale;
+ int result;
+
+ /*
+ * All variations on Unix require locale_t objects for LC_MONETARY and
+ * LC_NUMERIC. We'll set all locale categories, so that we can don't have
+ * to worry about POSIX's undefined behavior if LC_CTYPE's encoding
+ * doesn't match.
+ */
+ errno = ENOENT;
+ monetary_locale = newlocale(LC_ALL_MASK, lc_monetary, 0);
+ if (monetary_locale == 0)
+ return -1;
+ numeric_locale = newlocale(LC_ALL_MASK, lc_numeric, 0);
+ if (numeric_locale == 0)
+ {
+ freelocale(monetary_locale);
+ return -1;
+ }
+
+ memset(output, 0, sizeof(*output));
+#if defined(TRANSLATE_FROM_LANGINFO)
+ /* Copy from non-standard nl_langinfo_l() extended items. */
+ result = pg_localeconv_from_langinfo(output,
+ monetary_locale,
+ numeric_locale);
+#elif defined(HAVE_LOCALE_CONV_L)
+ /* Copy the LC_MONETARY members from a thread-safe lconv object. */
+ result = pg_localeconv_copy_members(output,
+ localeconv_l(monetary_locale),
+ LC_MONETARY);
+ if (result != 0)
+ goto exit;
+ /* Copy the LC_NUMERIC members from a thread-safe lconv object. */
+ result = pg_localeconv_copy_members(output,
+ localeconv_l(numeric_locale),
+ LC_NUMERIC);
+#else
+ /* We have nothing better than standard POSIX facilities. */
+ {
+ static pthread_mutex_t big_lock = PTHREAD_MUTEX_INITIALIZER;
+ locale_t save_locale;
+
+ pthread_mutex_lock(&big_lock);
+ /* Copy the LC_MONETARY members. */
+ save_locale = uselocale(monetary_locale);
+ result = pg_localeconv_copy_members(output,
+ localeconv(),
+ LC_MONETARY);
+ if (result == 0)
+ {
+ /* Copy the LC_NUMERIC members. */
+ uselocale(numeric_locale);
+ result = pg_localeconv_copy_members(output,
+ localeconv(),
+ LC_NUMERIC);
+ }
+ pthread_mutex_unlock(&big_lock);
+
+ uselocale(save_locale);
+ }
+#endif
+
+ freelocale(monetary_locale);
+ freelocale(numeric_locale);
+
+ return result;
+#endif
+}
--
2.46.0
[text/x-patch] v3-0002-Use-thread-safe-strftime_l-instead-of-strftime.patch (7.9K, ../../CA+hUKG+eEQ-+g2Ht-mho1o2xGX7v+GsuB4OQSUZv2U9S8huW8w@mail.gmail.com/3-v3-0002-Use-thread-safe-strftime_l-instead-of-strftime.patch)
download | inline diff:
From 23c9b4455fdd170c7f0e390ab1b38849607836df Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Wed, 14 Aug 2024 23:06:59 +1200
Subject: [PATCH v3 2/3] Use thread-safe strftime_l() instead of strftime().
This removes some setlocale() calls and a lot of commentary about how
dangerous that is. strftime_l() is from POSIX 2008, and on Windows we
use _wcsftime_l().
While here, adjust error message for strftime_l() failure: it does not
set errno, so no %m.
Reviewed-by:
Discussion: https://postgr.es/m/CA%2BhUKGJqVe0%2BPv9dvC9dSums_PXxGo9SWcxYAMBguWJUGbWz-A%40mail.gmail.com
---
src/backend/main/main.c | 5 +-
src/backend/utils/adt/pg_locale.c | 113 ++++++++----------------------
2 files changed, 31 insertions(+), 87 deletions(-)
diff --git a/src/backend/main/main.c b/src/backend/main/main.c
index 4672aab8378..4ffe1fd596e 100644
--- a/src/backend/main/main.c
+++ b/src/backend/main/main.c
@@ -121,10 +121,7 @@ main(int argc, char *argv[])
init_locale("LC_MESSAGES", LC_MESSAGES, "");
#endif
- /*
- * We keep these set to "C" always, except transiently in pg_locale.c; see
- * that file for explanations.
- */
+ /* We keep these set to "C" always. See pg_locale.c for explanation. */
init_locale("LC_MONETARY", LC_MONETARY, "C");
init_locale("LC_NUMERIC", LC_NUMERIC, "C");
init_locale("LC_TIME", LC_TIME, "C");
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index dd4ba9e0e89..1cf8efcc7b7 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -18,34 +18,13 @@
* LC_MESSAGES is settable at run time and will take effect
* immediately.
*
- * The other categories, LC_MONETARY, LC_NUMERIC, and LC_TIME are also
- * settable at run-time. However, we don't actually set those locale
- * categories permanently. This would have bizarre effects like no
- * longer accepting standard floating-point literals in some locales.
- * Instead, we only set these locale categories briefly when needed,
- * cache the required information obtained from localeconv() or
- * strftime(), and then set the locale categories back to "C".
+ * The other categories, LC_MONETARY, LC_NUMERIC, and LC_TIME are
+ * permanentaly set to "C", and then we use temporary locale_t
+ * objects when we need to look up locale data based on the GUCs
+ * of the same name. Information is cached when the GUCs change.
* The cached information is only used by the formatting functions
* (to_char, etc.) and the money type. For the user, this should all be
* transparent.
- *
- * !!! NOW HEAR THIS !!!
- *
- * We've been bitten repeatedly by this bug, so let's try to keep it in
- * mind in future: on some platforms, the locale functions return pointers
- * to static data that will be overwritten by any later locale function.
- * Thus, for example, the obvious-looking sequence
- * save = setlocale(category, NULL);
- * if (!setlocale(category, value))
- * fail = true;
- * setlocale(category, save);
- * DOES NOT WORK RELIABLY: on some platforms the second setlocale() call
- * will change the memory save is pointing at. To do this sort of thing
- * safely, you *must* pstrdup what setlocale returns the first time.
- *
- * The POSIX locale standard is available here:
- *
- * http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap07.html
*----------
*/
@@ -663,8 +642,8 @@ PGLC_localeconv(void)
* pg_strftime(), which isn't locale-aware and does not need to be replaced.
*/
static size_t
-strftime_win32(char *dst, size_t dstlen,
- const char *format, const struct tm *tm)
+strftime_l_win32(char *dst, size_t dstlen,
+ const char *format, const struct tm *tm, locale_t locale)
{
size_t len;
wchar_t wformat[8]; /* formats used below need 3 chars */
@@ -680,7 +659,7 @@ strftime_win32(char *dst, size_t dstlen,
elog(ERROR, "could not convert format string from UTF-8: error code %lu",
GetLastError());
- len = wcsftime(wbuf, MAX_L10N_DATA, wformat, tm);
+ len = _wcsftime_l(wbuf, MAX_L10N_DATA, wformat, tm, locale);
if (len == 0)
{
/*
@@ -701,8 +680,8 @@ strftime_win32(char *dst, size_t dstlen,
return len;
}
-/* redefine strftime() */
-#define strftime(a,b,c,d) strftime_win32(a,b,c,d)
+/* redefine strftime_l() */
+#define strftime_l(a,b,c,d,e) strftime_l_win32(a,b,c,d,e)
#endif /* WIN32 */
/*
@@ -743,10 +722,7 @@ cache_locale_time(void)
bool strftimefail = false;
int encoding;
int i;
- char *save_lc_time;
-#ifdef WIN32
- char *save_lc_ctype;
-#endif
+ locale_t locale;
/* did we do this already? */
if (CurrentLCTimeValid)
@@ -754,39 +730,21 @@ cache_locale_time(void)
elog(DEBUG3, "cache_locale_time() executed; locale: \"%s\"", locale_time);
- /*
- * As in PGLC_localeconv(), it's critical that we not throw error while
- * libc's locale settings have nondefault values. Hence, we just call
- * strftime() within the critical section, and then convert and save its
- * results afterwards.
- */
-
- /* Save prevailing value of time locale */
- save_lc_time = setlocale(LC_TIME, NULL);
- if (!save_lc_time)
- elog(ERROR, "setlocale(NULL) failed");
- save_lc_time = pstrdup(save_lc_time);
-
+ errno = ENOENT;
#ifdef WIN32
-
- /*
- * On Windows, it appears that wcsftime() internally uses LC_CTYPE, so we
- * must set it here. This code looks the same as what PGLC_localeconv()
- * does, but the underlying reason is different: this does NOT determine
- * the encoding we'll get back from strftime_win32().
- */
-
- /* Save prevailing value of ctype locale */
- save_lc_ctype = setlocale(LC_CTYPE, NULL);
- if (!save_lc_ctype)
- elog(ERROR, "setlocale(NULL) failed");
- save_lc_ctype = pstrdup(save_lc_ctype);
-
- /* use lc_time to set the ctype */
- setlocale(LC_CTYPE, locale_time);
+ locale = _create_locale(LC_ALL, locale_time);
+ if (locale == (locale_t) 0)
+ _dosmaperr(GetLastError());
+#else
+ locale = newlocale(LC_ALL_MASK, locale_time, (locale_t) 0);
#endif
+ if (locale == (locale_t) 0)
+ {
+ if (errno == ENOMEM)
+ elog(ERROR, "out of memory");
- setlocale(LC_TIME, locale_time);
+ elog(ERROR, "coud not create locale \"%s\": %m", locale_time);
+ }
/* We use times close to current time as data for strftime(). */
timenow = time(NULL);
@@ -809,10 +767,10 @@ cache_locale_time(void)
for (i = 0; i < 7; i++)
{
timeinfo->tm_wday = i;
- if (strftime(bufptr, MAX_L10N_DATA, "%a", timeinfo) <= 0)
+ if (strftime_l(bufptr, MAX_L10N_DATA, "%a", timeinfo, locale) <= 0)
strftimefail = true;
bufptr += MAX_L10N_DATA;
- if (strftime(bufptr, MAX_L10N_DATA, "%A", timeinfo) <= 0)
+ if (strftime_l(bufptr, MAX_L10N_DATA, "%A", timeinfo, locale) <= 0)
strftimefail = true;
bufptr += MAX_L10N_DATA;
}
@@ -822,37 +780,26 @@ cache_locale_time(void)
{
timeinfo->tm_mon = i;
timeinfo->tm_mday = 1; /* make sure we don't have invalid date */
- if (strftime(bufptr, MAX_L10N_DATA, "%b", timeinfo) <= 0)
+ if (strftime_l(bufptr, MAX_L10N_DATA, "%b", timeinfo, locale) <= 0)
strftimefail = true;
bufptr += MAX_L10N_DATA;
- if (strftime(bufptr, MAX_L10N_DATA, "%B", timeinfo) <= 0)
+ if (strftime_l(bufptr, MAX_L10N_DATA, "%B", timeinfo, locale) <= 0)
strftimefail = true;
bufptr += MAX_L10N_DATA;
}
- /*
- * Restore the prevailing locale settings; as in PGLC_localeconv(),
- * failure to do so is fatal.
- */
#ifdef WIN32
- if (!setlocale(LC_CTYPE, save_lc_ctype))
- elog(FATAL, "failed to restore LC_CTYPE to \"%s\"", save_lc_ctype);
+ _free_locale(locale);
+#else
+ freelocale(locale);
#endif
- if (!setlocale(LC_TIME, save_lc_time))
- elog(FATAL, "failed to restore LC_TIME to \"%s\"", save_lc_time);
/*
* At this point we've done our best to clean up, and can throw errors, or
* call functions that might throw errors, with a clean conscience.
*/
if (strftimefail)
- elog(ERROR, "strftime() failed: %m");
-
- /* Release the pstrdup'd locale names */
- pfree(save_lc_time);
-#ifdef WIN32
- pfree(save_lc_ctype);
-#endif
+ elog(ERROR, "strftime_l() failed");
#ifndef WIN32
--
2.46.0
[text/x-patch] v3-0003-Remove-setlocale-calls-from-check_locale.patch (9.8K, ../../CA+hUKG+eEQ-+g2Ht-mho1o2xGX7v+GsuB4OQSUZv2U9S8huW8w@mail.gmail.com/4-v3-0003-Remove-setlocale-calls-from-check_locale.patch)
download | inline diff:
From ad80a4097ba76acbe1434208a060cb271f7e6155 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Thu, 15 Aug 2024 16:45:27 +1200
Subject: [PATCH v3 3/3] Remove setlocale() calls from check_locale().
Validate locale names with newlocale() or _create_locale() instead, to
avoid clobbering global state.
This removes the previous assumption that it's useful to canonicalize
locale names with setlocale(), which wasn't really true, at least on any
known Unix (thanks to Tom Lane for this observation). Two kinds of name
transformation are still useful:
1. "" means use the contents of LC_xxx environment variables, which we
can easily look up ourselves instead of asking setlocale() to do it.
(We set them ourselves in pg_perm_setlocale() early in main().)
2. Windows setlocale() apparently does some transformations (for
example the EDB installer passes it "Language,Country" and it returns
"Language_Country.CodePage"). While all such locale names are an
unstable mess and deprecated in favor of BCP 47 on that OS since ~2007,
we want to keep supporting that for a bit longer, so create a
canonicalization function in win32setlocale.c that is careful to be
thread-safe, so that it doesn't get in the way of our plan to make the
backend potentially thread-safe.
Reviewed-by:
Discussion: https://postgr.es/m/CA%2BhUKGJqVe0%2BPv9dvC9dSums_PXxGo9SWcxYAMBguWJUGbWz-A%40mail.gmail.com
Discussion: https://postgr.es/m/CA%2BhUKGK57sgUYKO03jB4VarTsswfMyScFAyJpVnYD8c%2Bg12_mg%40mail.gmail.com
---
src/backend/utils/adt/pg_locale.c | 193 ++++++++++++++++++++++--------
src/include/port/win32_port.h | 2 +
src/port/win32setlocale.c | 48 ++++++++
3 files changed, 190 insertions(+), 53 deletions(-)
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 1cf8efcc7b7..b11d3ac81e7 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -190,6 +190,65 @@ wcstombs_l(char *dest, const wchar_t *src, size_t n, locale_t loc)
}
#endif
+/*
+ * The category names as strings. These are the names of the environment
+ * variables that define the server locale environment. We always unset
+ * LC_ALL, so we only need the actual categories.
+ */
+static const char *
+get_lc_category_name(int category)
+{
+ switch (category)
+ {
+ case LC_COLLATE:
+ return "LC_COLLATE";
+ case LC_CTYPE:
+ return "LC_CTYPE";
+#ifdef LC_MESSAGES
+ case LC_MESSAGES:
+ return "LC_MESSAGES";
+#endif
+ case LC_MONETARY:
+ return "LC_MONETARY";
+ case LC_NUMERIC:
+ return "LC_NUMERIC";
+ case LC_TIME:
+ return "LC_TIME";
+ default:
+ return NULL;
+ };
+};
+
+#ifndef WIN32
+/*
+ * The newlocale() function needs LC_xxx_MASK, but sometimes we have LC_xxx,
+ * and POSIX doesn't offer a way to translate.
+ */
+static int
+get_lc_category_mask(int category)
+{
+ switch (category)
+ {
+ case LC_COLLATE:
+ return LC_COLLATE_MASK;
+ case LC_CTYPE:
+ return LC_CTYPE_MASK;
+#ifdef LC_MESSAGES
+ case LC_MESSAGES:
+ return LC_MESSAGES_MASK;
+#endif
+ case LC_MONETARY:
+ return LC_MONETARY_MASK;
+ case LC_NUMERIC:
+ return LC_NUMERIC_MASK;
+ case LC_TIME:
+ return LC_TIME_MASK;
+ default:
+ return 0;
+ };
+}
+#endif
+
/*
* pg_perm_setlocale
*
@@ -257,38 +316,9 @@ pg_perm_setlocale(int category, const char *locale)
#endif
}
- switch (category)
- {
- case LC_COLLATE:
- envvar = "LC_COLLATE";
- break;
- case LC_CTYPE:
- envvar = "LC_CTYPE";
- break;
-#ifdef LC_MESSAGES
- case LC_MESSAGES:
- envvar = "LC_MESSAGES";
-#ifdef WIN32
- result = IsoLocaleName(locale);
- if (result == NULL)
- result = (char *) locale;
- elog(DEBUG3, "IsoLocaleName() executed; locale: \"%s\"", result);
-#endif /* WIN32 */
- break;
-#endif /* LC_MESSAGES */
- case LC_MONETARY:
- envvar = "LC_MONETARY";
- break;
- case LC_NUMERIC:
- envvar = "LC_NUMERIC";
- break;
- case LC_TIME:
- envvar = "LC_TIME";
- break;
- default:
- elog(FATAL, "unrecognized LC category: %d", category);
- return NULL; /* keep compiler quiet */
- }
+ envvar = get_lc_category_name(category);
+ if (!envvar)
+ elog(FATAL, "unrecognized LC category: %d", category);
if (setenv(envvar, result, 1) != 0)
return NULL;
@@ -302,40 +332,97 @@ pg_perm_setlocale(int category, const char *locale)
*
* If successful, and canonname isn't NULL, a palloc'd copy of the locale's
* canonical name is stored there. This is especially useful for figuring out
- * what locale name "" means (ie, the server environment value). (Actually,
- * it seems that on most implementations that's the only thing it's good for;
- * we could wish that setlocale gave back a canonically spelled version of
- * the locale name, but typically it doesn't.)
+ * what locale name "" means (ie, the server environment value). On Windows,
+ * it also gives a canonically spelled version of the locale name, when using
+ * traditional Windows (pre-BCP 47) locale names.
*/
bool
check_locale(int category, const char *locale, char **canonname)
{
- char *save;
- char *res;
+ locale_t loc;
if (canonname)
*canonname = NULL; /* in case of failure */
- save = setlocale(category, NULL);
- if (!save)
- return false; /* won't happen, we hope */
+ if (locale[0] == 0)
+ {
+ /*
+ * Caller asked for "", meaning the "native environment" in POSIX
+ * terminology. This means the LC_XXX environment variables, which
+ * pg_perm_setlocale() sets, so we can get them directly from there.
+ * That is exactly what POSIX setlocale() is required to do for "",
+ * except that it would also check "LC_ALL" and "LANG", and we unset
+ * the former and the latter has lower priority than the category
+ * names.
+ */
+ const char *envvar = get_lc_category_name(category);
- /* save may be pointing at a modifiable scratch variable, see above. */
- save = pstrdup(save);
+ if (!envvar)
+ return false;
+
+ locale = getenv(envvar);
+ if (locale)
+ return false;
+ }
+
+ /*
+ * See if we can open it. Unfortunately we can't always distinguish
+ * out-of-memory from invalid locale name.
+ */
+ errno = ENOENT;
+#ifdef WIN32
+ loc = _create_locale(category, locale);
+ if (loc == (locale_t) 0)
+ _dosmaperr(GetLastError());
+#else
+ loc = newlocale(get_lc_category_mask(category), locale, (locale_t) 0);
+#endif
+ if (loc == (locale_t) 0)
+ {
+ if (errno == ENOMEM)
+ elog(ERROR, "out of memory");
- /* set the locale with setlocale, to see if it accepts it. */
- res = setlocale(category, locale);
+ /* Otherwise assume the locale doesn't exist. */
+ return false;
+ }
+#ifdef WIN32
+ _free_locale(loc);
+#else
+ freelocale(loc);
+#endif
- /* save canonical name if requested. */
- if (res && canonname)
- *canonname = pstrdup(res);
+ if (canonname)
+ {
+#ifdef WIN32
+ char *canonicalized;
- /* restore old value. */
- if (!setlocale(category, save))
- elog(WARNING, "failed to restore old locale \"%s\"", save);
- pfree(save);
+ /*
+ * On Windows, we pass it through setlocale() in thread-local mode,
+ * which gives a canonical version of the name.
+ */
+ canonicalized = pgwin32_canonicalize_locale_name(category, locale);
+ if (!canonicalized)
+ return false;
+ /* Copy from malloc'd memory to palloc'd memory. */
+ *canonname = palloc_extended(strlen(canonicalized) + 1,
+ MCXT_ALLOC_NO_OOM);
+ if (!*canonname)
+ {
+ free(canonicalized);
+ return false;
+ }
+ strcpy(*canonname, canonicalized);
+ free(canonicalized);
+#else
+ /* On Unix there is no such concept, so just copy the name verbatim. */
+ *canonname = palloc_extended(strlen(locale) + 1, MCXT_ALLOC_NO_OOM);
+ if (!*canonname)
+ return false;
+ strcpy(*canonname, locale);
+#endif
+ }
- return (res != NULL);
+ return true;
}
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index 7ffe5891c69..30bcf14812f 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -471,6 +471,8 @@ extern char *pgwin32_setlocale(int category, const char *locale);
#define setlocale(a,b) pgwin32_setlocale(a,b)
+extern char *pgwin32_canonicalize_locale_name(int category,
+ const char *locale);
/* In backend/port/win32/signal.c */
extern PGDLLIMPORT volatile int pg_signal_queue;
diff --git a/src/port/win32setlocale.c b/src/port/win32setlocale.c
index 9e2ab8cc3ad..e45b76c9fca 100644
--- a/src/port/win32setlocale.c
+++ b/src/port/win32setlocale.c
@@ -191,3 +191,51 @@ pgwin32_setlocale(int category, const char *locale)
return result;
}
+
+/*
+ * Returns a malloc'd copy of the name that Windows returns when you set a
+ * locale, or NULL on error. Since this function is usually called after the
+ * name has been validated, failure here likely implies out-of-memory.
+ *
+ * Modern BCP 47 locale names such as "en-US" are not expected to be changed by
+ * this function, but for the older deprecated "English_United States.1252"
+ * format, several variations seem to be accepted and converted to that form.
+ *
+ * This function is thread-safe, because it puts setlocale() into thread-local
+ * mode temporarily. It uses wchar_t for save-and-restore, to avoid problems
+ * restoring the old locale if the code page changes.
+ */
+char *
+pgwin32_canonicalize_locale_name(int category, const char *locale)
+{
+ wchar_t *save_locale = NULL;
+ int save_config_thread_locale;
+ char *canonical;
+ char *result = NULL;
+
+ save_config_thread_locale = _configthreadlocale(_ENABLE_PER_THREAD_LOCALE);
+
+ save_locale = _wsetlocale(category, NULL);
+ if (!save_locale || !(save_locale = wcsdup(save_locale)))
+ goto exit;
+
+ canonical = setlocale(category, locale);
+ if (!canonical)
+ goto exit;
+
+ result = malloc(strlen(canonical) + 1);
+ if (!result)
+ goto exit;
+ strcpy(result, canonical);
+
+exit:
+ /* Restore everything we changed. */
+ if (save_locale)
+ {
+ _wsetlocale(category, save_locale);
+ free(save_locale);
+ }
+ _configthreadlocale(save_config_thread_locale);
+
+ return result;
+}
--
2.46.0
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Thread-safe nl_langinfo() and localeconv()
@ 2024-08-15 11:11 Heikki Linnakangas <[email protected]>
parent: Thomas Munro <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Heikki Linnakangas @ 2024-08-15 11:11 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: pgsql-hackers
On 15/08/2024 11:03, Thomas Munro wrote:
> Here's a new patch to add to this pile, this time for check_locale().
> I also improved the error handling and comments in the other patches.
There's a similar function in initdb, check_locale_name. I wonder if
that could reuse the same code.
I wonder if it would be more clear to split this into three functions:
/*
* Get the name of the locale in "native environment",
* like setlocale(category, NULL) does
*/
char *get_native_locale(int category);
/*
* Return true if 'locale' is valid locale name for 'category
*/
bool check_locale(int category, const char *locale);
/*
* Return a canonical name of given locale
*/
char *canonicalize_locale(int category, const char *locale);
> result = malloc(strlen(canonical) + 1);
> if (!result)
> goto exit;
> strcpy(result, canonical);
Could use "result = strdup(canonical)" here. Or even better, could we
use palloc()/pstrdup() here, and save the caller the trouble to copy it?
--
Heikki Linnakangas
Neon (https://neon.tech)
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Thread-safe nl_langinfo() and localeconv()
@ 2024-08-16 00:48 Thomas Munro <[email protected]>
parent: Heikki Linnakangas <[email protected]>
0 siblings, 1 reply; 51+ messages in thread
From: Thomas Munro @ 2024-08-16 00:48 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: pgsql-hackers
On Thu, Aug 15, 2024 at 11:11 PM Heikki Linnakangas <[email protected]> wrote:
> There's a similar function in initdb, check_locale_name. I wonder if
> that could reuse the same code.
Thanks, between this comment and some observations from Peter E and
Tom, I think I have a better plan now. I think they should *not*
match, and a comment saying so should be deleted. In the backend, we
should do neither ""-expansion (ie getenv("LC_...") whether direct or
indirect) nor canonicalisation (of Windows' deprecated pre-BCP 47
locale names), making this v4 patch extremely simple.
1. CREATE DATABASE doesn't really need to accept LOCALE = ''. What
is the point? It's not documented or desirable behavior AFAIK. If
you like defaults you can just not provide a locale at all and get the
template database's (which often comes from initdb, which often uses
the server environment). That behavior was already inconsistent with
CREATE COLLATION. So I think we should just reject "" in the backend
check_locale().
2. A similar argument applies to Windows canonicalisation. CREATE
COLLATION isn't doing it. CREATE DATABASE is, but again, what is the
point? See previous.
(I also think that initdb should use a different mechanism for finding
the native locale on Windows, but I already have a CF #3772 for that,
ie migration plan for BCP 47 and native UTF-8 on Windows, but I don't
want *this* thread to get blocked by our absence of Windows
reviewers/testers, so let's not tangle that up with this thread-safety
expedition.)
To show a concrete example of commands no longer accepted with this
version, because they call check_locale():
postgres=# set lc_monetary = '';
ERROR: invalid value for parameter "lc_monetary": ""
postgres=# create database db2 locale = '';
ERROR: invalid LC_COLLATE locale name: ""
HINT: If the locale name is specific to ICU, use ICU_LOCALE.
Does anyone see a problem with that?
I do see a complication for CREATE COLLATION, though. It doesn't call
check_locale(), is not changed in this patch, and still accepts ''.
Reasoning: There may be systems with '' in their pg_collation catalog
in the wild, since we never canonicalised with setlocale(), so it
might create some kind of unforeseen dump/restore/upgrade hazard if we
just ban '' outright, I just don't know what yet.
There is no immediate problem, ie there is no setlocale() to excise,
for *this* project. Longer term, we can't actually continue to allow
'' in COLLATION objects, though: that tells newlocale() to call
getenv(), which may be technically OK in a multi-threaded program
(that never calls setenv()), but is hardly desirable. But it will
also give the wrong result, if we pursue the plan that Jeff and I
discussed: we'll stop doing setenv("LC_COLLATE", datacollate) and
setenv("LC_CTYPE", datctype) in postinit.c (see pg_perm_setlocale()
calls). So I think any pg_collation catalog entries holding '' need
to be translated to datcollate/datctype... somewhere. I just don't
know where yet and don't want to tackle that in the same patch.
Attachments:
[text/x-patch] v4-0001-Provide-thread-safe-pg_localeconv_r.patch (22.5K, ../../CA+hUKGLqLxaNXN8mSqetYpD0QHWimHWZbi6KOfP=hfRvBfA9iw@mail.gmail.com/2-v4-0001-Provide-thread-safe-pg_localeconv_r.patch)
download | inline diff:
From 4831ff4373b9d713c78e303d9758de347aadfc2f Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Tue, 13 Aug 2024 14:15:54 +1200
Subject: [PATCH v4 1/3] Provide thread-safe pg_localeconv_r().
This involves four different implementation strategies:
1. For Windows, we now require _configthreadlocale() to be available
and work, and the documentation says that the object returned by
localeconv() is in thread-local memory.
2. For glibc, we translate to nl_langinfo_l() calls, because it offers
the same information that way as an extension, and that API is
thread-safe.
3. For macOS/*BSD, use localeconv_l(), which is thread-safe.
4. For everything else, use uselocale() to set the locale for the
thread, and use a big ugly lock to defend against the returned object
being concurrently clobbered. In practice this currently means only
Solaris.
The new call is used in pg_locale.c, replacing calls to setlocale() and
localeconv().
This patch adds a hard requirement on Windows' _configthreadlocale().
In the past there were said to be MinGW systems that didn't have it, or
had it but it didn't work. As far as I can tell, CI (optional MinGW
task + mingw cross build warning task) and build farm (fairywren msys)
should be happy with this. Fingers crossed. (There are places that use
configure checks for that in ECPG; other proposed patches would remove
those later.)
Reviewed-by: Heikki Linnakangas <[email protected]>
Discussion: https://postgr.es/m/CA%2BhUKGJqVe0%2BPv9dvC9dSums_PXxGo9SWcxYAMBguWJUGbWz-A%40mail.gmail.com
---
configure | 2 +-
configure.ac | 1 +
meson.build | 1 +
src/backend/utils/adt/pg_locale.c | 128 ++---------
src/include/pg_config.h.in | 3 +
src/include/port.h | 6 +
src/port/Makefile | 1 +
src/port/meson.build | 1 +
src/port/pg_localeconv_r.c | 367 ++++++++++++++++++++++++++++++
9 files changed, 402 insertions(+), 108 deletions(-)
create mode 100644 src/port/pg_localeconv_r.c
diff --git a/configure b/configure
index 2abbeb27944..60dcf1e436e 100755
--- a/configure
+++ b/configure
@@ -15232,7 +15232,7 @@ fi
LIBS_including_readline="$LIBS"
LIBS=`echo "$LIBS" | sed -e 's/-ledit//g' -e 's/-lreadline//g'`
-for ac_func in backtrace_symbols copyfile copy_file_range getifaddrs getpeerucred inet_pton kqueue mbstowcs_l memset_s posix_fallocate ppoll pthread_is_threaded_np setproctitle setproctitle_fast strchrnul strsignal syncfs sync_file_range uselocale wcstombs_l
+for ac_func in backtrace_symbols copyfile copy_file_range getifaddrs getpeerucred inet_pton localeconv_l kqueue mbstowcs_l memset_s posix_fallocate ppoll pthread_is_threaded_np setproctitle setproctitle_fast strchrnul strsignal syncfs sync_file_range uselocale wcstombs_l
do :
as_ac_var=`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh`
ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var"
diff --git a/configure.ac b/configure.ac
index c46ed2c591a..59e51a74629 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1735,6 +1735,7 @@ AC_CHECK_FUNCS(m4_normalize([
getifaddrs
getpeerucred
inet_pton
+ localeconv_l
kqueue
mbstowcs_l
memset_s
diff --git a/meson.build b/meson.build
index cd711c6d018..028a14547aa 100644
--- a/meson.build
+++ b/meson.build
@@ -2675,6 +2675,7 @@ func_checks = [
['inet_aton'],
['inet_pton'],
['kqueue'],
+ ['localeconv_l'],
['mbstowcs_l'],
['memset_s'],
['mkdtemp'],
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index cd3661e7279..dd4ba9e0e89 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -543,12 +543,8 @@ PGLC_localeconv(void)
static struct lconv CurrentLocaleConv;
static bool CurrentLocaleConvAllocated = false;
struct lconv *extlconv;
- struct lconv worklconv;
- char *save_lc_monetary;
- char *save_lc_numeric;
-#ifdef WIN32
- char *save_lc_ctype;
-#endif
+ struct lconv tmp;
+ struct lconv worklconv = {0};
/* Did we do it already? */
if (CurrentLocaleConvValid)
@@ -562,77 +558,21 @@ PGLC_localeconv(void)
}
/*
- * This is tricky because we really don't want to risk throwing error
- * while the locale is set to other than our usual settings. Therefore,
- * the process is: collect the usual settings, set locale to special
- * setting, copy relevant data into worklconv using strdup(), restore
- * normal settings, convert data to desired encoding, and finally stash
- * the collected data in CurrentLocaleConv. This makes it safe if we
- * throw an error during encoding conversion or run out of memory anywhere
- * in the process. All data pointed to by struct lconv members is
- * allocated with strdup, to avoid premature elog(ERROR) and to allow
- * using a single cleanup routine.
+ * Use thread-safe method of obtaining a copy of lconv from the operating
+ * system.
*/
- memset(&worklconv, 0, sizeof(worklconv));
-
- /* Save prevailing values of monetary and numeric locales */
- save_lc_monetary = setlocale(LC_MONETARY, NULL);
- if (!save_lc_monetary)
- elog(ERROR, "setlocale(NULL) failed");
- save_lc_monetary = pstrdup(save_lc_monetary);
-
- save_lc_numeric = setlocale(LC_NUMERIC, NULL);
- if (!save_lc_numeric)
- elog(ERROR, "setlocale(NULL) failed");
- save_lc_numeric = pstrdup(save_lc_numeric);
-
-#ifdef WIN32
-
- /*
- * The POSIX standard explicitly says that it is undefined what happens if
- * LC_MONETARY or LC_NUMERIC imply an encoding (codeset) different from
- * that implied by LC_CTYPE. In practice, all Unix-ish platforms seem to
- * believe that localeconv() should return strings that are encoded in the
- * codeset implied by the LC_MONETARY or LC_NUMERIC locale name. Hence,
- * once we have successfully collected the localeconv() results, we will
- * convert them from that codeset to the desired server encoding.
- *
- * Windows, of course, resolutely does things its own way; on that
- * platform LC_CTYPE has to match LC_MONETARY/LC_NUMERIC to get sane
- * results. Hence, we must temporarily set that category as well.
- */
-
- /* Save prevailing value of ctype locale */
- save_lc_ctype = setlocale(LC_CTYPE, NULL);
- if (!save_lc_ctype)
- elog(ERROR, "setlocale(NULL) failed");
- save_lc_ctype = pstrdup(save_lc_ctype);
-
- /* Here begins the critical section where we must not throw error */
-
- /* use numeric to set the ctype */
- setlocale(LC_CTYPE, locale_numeric);
-#endif
-
- /* Get formatting information for numeric */
- setlocale(LC_NUMERIC, locale_numeric);
- extlconv = localeconv();
-
- /* Must copy data now in case setlocale() overwrites it */
+ if (pg_localeconv_r(locale_monetary,
+ locale_numeric,
+ &tmp) != 0)
+ elog(ERROR,
+ "could not get lconv for LC_MONETARY = \"%s\", LC_NUMERIC = \"%s\": %m",
+ locale_monetary, locale_numeric);
+
+ /* Must copy data now now so we can re-encode it. */
+ extlconv = &tmp;
worklconv.decimal_point = strdup(extlconv->decimal_point);
worklconv.thousands_sep = strdup(extlconv->thousands_sep);
worklconv.grouping = strdup(extlconv->grouping);
-
-#ifdef WIN32
- /* use monetary to set the ctype */
- setlocale(LC_CTYPE, locale_monetary);
-#endif
-
- /* Get formatting information for monetary */
- setlocale(LC_MONETARY, locale_monetary);
- extlconv = localeconv();
-
- /* Must copy data now in case setlocale() overwrites it */
worklconv.int_curr_symbol = strdup(extlconv->int_curr_symbol);
worklconv.currency_symbol = strdup(extlconv->currency_symbol);
worklconv.mon_decimal_point = strdup(extlconv->mon_decimal_point);
@@ -650,45 +590,19 @@ PGLC_localeconv(void)
worklconv.p_sign_posn = extlconv->p_sign_posn;
worklconv.n_sign_posn = extlconv->n_sign_posn;
- /*
- * Restore the prevailing locale settings; failure to do so is fatal.
- * Possibly we could limp along with nondefault LC_MONETARY or LC_NUMERIC,
- * but proceeding with the wrong value of LC_CTYPE would certainly be bad
- * news; and considering that the prevailing LC_MONETARY and LC_NUMERIC
- * are almost certainly "C", there's really no reason that restoring those
- * should fail.
- */
-#ifdef WIN32
- if (!setlocale(LC_CTYPE, save_lc_ctype))
- elog(FATAL, "failed to restore LC_CTYPE to \"%s\"", save_lc_ctype);
-#endif
- if (!setlocale(LC_MONETARY, save_lc_monetary))
- elog(FATAL, "failed to restore LC_MONETARY to \"%s\"", save_lc_monetary);
- if (!setlocale(LC_NUMERIC, save_lc_numeric))
- elog(FATAL, "failed to restore LC_NUMERIC to \"%s\"", save_lc_numeric);
+ /* Free the contents of the object populated by pg_localeconv_r(). */
+ pg_localeconv_free(&tmp);
+
+ /* If any of the preceding strdup calls failed, complain now. */
+ if (!struct_lconv_is_valid(&worklconv))
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory")));
- /*
- * At this point we've done our best to clean up, and can call functions
- * that might possibly throw errors with a clean conscience. But let's
- * make sure we don't leak any already-strdup'd fields in worklconv.
- */
PG_TRY();
{
int encoding;
- /* Release the pstrdup'd locale names */
- pfree(save_lc_monetary);
- pfree(save_lc_numeric);
-#ifdef WIN32
- pfree(save_lc_ctype);
-#endif
-
- /* If any of the preceding strdup calls failed, complain now. */
- if (!struct_lconv_is_valid(&worklconv))
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory")));
-
/*
* Now we must perform encoding conversion from whatever's associated
* with the locales into the database encoding. If we can't identify
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 979925cc2e2..f3db06d155f 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -280,6 +280,9 @@
/* Define to 1 if you have the `zstd' library (-lzstd). */
#undef HAVE_LIBZSTD
+/* Define to 1 if you have the `localeconv_l' function. */
+#undef HAVE_LOCALECONV_L
+
/* Define to 1 if `long int' works and is 64 bits. */
#undef HAVE_LONG_INT_64
diff --git a/src/include/port.h b/src/include/port.h
index c7400052675..ac0cff79fc6 100644
--- a/src/include/port.h
+++ b/src/include/port.h
@@ -465,6 +465,12 @@ extern void *bsearch_arg(const void *key, const void *base0,
int (*compar) (const void *, const void *, void *),
void *arg);
+/* port/pg_localeconv_r.c */
+extern int pg_localeconv_r(const char *lc_monetary,
+ const char *lc_numeric,
+ struct lconv *output);
+extern void pg_localeconv_free(struct lconv *lconv);
+
/* port/chklocale.c */
extern int pg_get_encoding_from_locale(const char *ctype, bool write_message);
diff --git a/src/port/Makefile b/src/port/Makefile
index db7c02117b0..f24d2dbc138 100644
--- a/src/port/Makefile
+++ b/src/port/Makefile
@@ -45,6 +45,7 @@ OBJS = \
noblock.o \
path.o \
pg_bitutils.o \
+ pg_localeconv_r.o \
pg_strong_random.o \
pgcheckdir.o \
pgmkdirp.o \
diff --git a/src/port/meson.build b/src/port/meson.build
index ff54b7b53e9..9d4c4018523 100644
--- a/src/port/meson.build
+++ b/src/port/meson.build
@@ -7,6 +7,7 @@ pgport_sources = [
'noblock.c',
'path.c',
'pg_bitutils.c',
+ 'pg_localeconv_r.c',
'pg_strong_random.c',
'pgcheckdir.c',
'pgmkdirp.c',
diff --git a/src/port/pg_localeconv_r.c b/src/port/pg_localeconv_r.c
new file mode 100644
index 00000000000..efb98cd127d
--- /dev/null
+++ b/src/port/pg_localeconv_r.c
@@ -0,0 +1,367 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_localeconv_r.c
+ * Thread-safe implementations of localeconv()
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/port/pg_localeconv_r.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "c.h"
+
+#if !defined(WIN32)
+#include <langinfo.h>
+#include <pthread.h>
+#endif
+
+#include <limits.h>
+
+#ifdef MON_THOUSANDS_SEP
+/*
+ * One of glibc's extended langinfo items detected. Assume that the full set
+ * is present, which means we can use nl_langinfo_l() instead of localeconv().
+ */
+#define TRANSLATE_FROM_LANGINFO
+#endif
+
+struct lconv_member_info
+{
+ bool is_string;
+ int category;
+ size_t offset;
+#ifdef TRANSLATE_FROM_LANGINFO
+ nl_item item;
+#endif
+};
+
+/* Some macros to declare the lconv members compactly. */
+#ifdef TRANSLATE_FROM_LANGINFO
+#define LCONV_M(is_string, category, name, item) \
+ { is_string, category, offsetof(struct lconv, name), item }
+#else
+#define LCONV_M(is_string, category, name, item) \
+ { is_string, category, offsetof(struct lconv, name) }
+#endif
+#define LCONV_S(c, n, i) LCONV_M(true, c, n, i)
+#define LCONV_C(c, n, i) LCONV_M(false, c, n, i)
+
+/*
+ * The work of populating lconv objects is driven by this table. Since we
+ * tolerate non-matching encodings in LC_NUMERIC and LC_MONETARY, we have to
+ * call the underlying OS routine multiple times, with the correct locales.
+ * The first column of this table says which locale applies to each struct
+ * member. The second column is the name of the struct member. The third
+ * column is the name of the nl_item, if translating from nl_langinfo_l() (it's
+ * always the member name, in upper case).
+ */
+const static struct lconv_member_info table[] = {
+ /* String fields. */
+ LCONV_S(LC_NUMERIC, decimal_point, DECIMAL_POINT),
+ LCONV_S(LC_NUMERIC, thousands_sep, THOUSANDS_SEP),
+ LCONV_S(LC_NUMERIC, grouping, GROUPING),
+ LCONV_S(LC_MONETARY, int_curr_symbol, INT_CURR_SYMBOL),
+ LCONV_S(LC_MONETARY, currency_symbol, CURRENCY_SYMBOL),
+ LCONV_S(LC_MONETARY, mon_decimal_point, MON_DECIMAL_POINT),
+ LCONV_S(LC_MONETARY, mon_thousands_sep, MON_THOUSANDS_SEP),
+ LCONV_S(LC_MONETARY, mon_grouping, MON_GROUPING),
+ LCONV_S(LC_MONETARY, positive_sign, POSITIVE_SIGN),
+ LCONV_S(LC_MONETARY, negative_sign, NEGATIVE_SIGN),
+
+ /* Character fields. */
+ LCONV_C(LC_MONETARY, int_frac_digits, INT_FRAC_DIGITS),
+ LCONV_C(LC_MONETARY, frac_digits, FRAC_DIGITS),
+ LCONV_C(LC_MONETARY, p_cs_precedes, P_CS_PRECEDES),
+ LCONV_C(LC_MONETARY, p_sep_by_space, P_SEP_BY_SPACE),
+ LCONV_C(LC_MONETARY, n_cs_precedes, N_CS_PRECEDES),
+ LCONV_C(LC_MONETARY, n_sep_by_space, N_SEP_BY_SPACE),
+ LCONV_C(LC_MONETARY, p_sign_posn, P_SIGN_POSN),
+ LCONV_C(LC_MONETARY, n_sign_posn, N_SIGN_POSN),
+};
+
+static inline char **
+lconv_string_member(struct lconv *lconv, int i)
+{
+ return (char **) ((char *) lconv + table[i].offset);
+}
+
+static inline char *
+lconv_char_member(struct lconv *lconv, int i)
+{
+ return (char *) lconv + table[i].offset;
+}
+
+/*
+ * Free the members of a struct lconv populated by pg_localeconv_r(). The
+ * struct itself is in storage provided by the caller of pg_localeconv_r().
+ */
+void
+pg_localeconv_free(struct lconv *lconv)
+{
+ for (int i = 0; i < lengthof(table); ++i)
+ if (table[i].is_string)
+ free(*lconv_string_member(lconv, i));
+}
+
+#ifdef TRANSLATE_FROM_LANGINFO
+/*
+ * Fill in struct lconv members using the equivalent nl_langinfo_l() items.
+ */
+static int
+pg_localeconv_from_langinfo(struct lconv *dst,
+ locale_t monetary_locale,
+ locale_t numeric_locale)
+{
+ for (int i = 0; i < lengthof(table); ++i)
+ {
+ locale_t locale;
+
+ locale = table[i].category == LC_NUMERIC ?
+ numeric_locale : monetary_locale;
+
+ if (table[i].is_string)
+ {
+ char *string;
+
+ string = nl_langinfo_l(table[i].item, locale);
+ if (!(string = strdup(string)))
+ {
+ pg_localeconv_free(dst);
+ errno = ENOMEM;
+ return -1;
+ }
+ *lconv_string_member(dst, i) = string;
+ }
+ else
+ {
+ *lconv_char_member(dst, i) =
+ *nl_langinfo_l(table[i].item, locale);
+ }
+ }
+
+ return 0;
+}
+#else
+/*
+ * Copy members from a given category. Note that you have to call this twice
+ * to copy the LC_MONETARY and then LC_NUMERIC members.
+ */
+static int
+pg_localeconv_copy_members(struct lconv *dst,
+ struct lconv *src,
+ int category)
+{
+ for (int i = 0; i < lengthof(table); ++i)
+ {
+ if (table[i].category != category)
+ continue;
+
+ if (table[i].is_string)
+ {
+ char *string;
+
+ string = *lconv_string_member(src, i);
+ if (!(string = strdup(string)))
+ {
+ pg_localeconv_free(dst);
+ errno = ENOMEM;
+ return -1;
+ }
+ *lconv_string_member(dst, i) = string;
+ }
+ else
+ {
+ *lconv_char_member(dst, i) = *lconv_char_member(src, i);
+ }
+ }
+
+ return 0;
+}
+#endif
+
+/*
+ * A thread-safe routine to get a copy of the lconv struct for a given
+ * LC_NUMERIC and LC_MONETARY. Different approaches are used on different
+ * OSes, because the standard interface is so multi-threading unfriendly.
+ *
+ * 1. On Windows, there is no uselocale(), but there is a way to put
+ * setlocale() into a thread-local mode temporarily. Its localeconv() is
+ * documented as returning a pointer to thread-local storage, so we don't have
+ * to worry about concurrent callers.
+ *
+ * 2. On Glibc, as an extension, all the information required to populate
+ * struct lconv is also available via nl_langpath_l(), which is thread-safe.
+ *
+ * 3. On macOS and *BSD, there is localeconv_l(), so we can create a temporary
+ * locale_t to pass in, and the result is a pointer to storage associated with
+ * the locale_t so we control its lifetime and we don't have to worry about
+ * concurrent calls clobbering it.
+ *
+ * 4. Otherwise, we wrap plain old localeconv() in uselocale() to avoid
+ * touching the global locale, but the output buffer is allowed by the standard
+ * to be overwritten by concurrent calls to localeconv(). We protect against
+ * _this_ function doing that with a Big Lock, but there isn't much we can do
+ * about code outside our tree that might call localeconv(), given such a poor
+ * interface.
+ *
+ * The POSIX standard explicitly says that it is undefined what happens if
+ * LC_MONETARY or LC_NUMERIC imply an encoding (codeset) different from that
+ * implied by LC_CTYPE. In practice, all Unix-ish platforms seem to believe
+ * that localeconv() should return strings that are encoded in the codeset
+ * implied by the LC_MONETARY or LC_NUMERIC locale name. On Windows, LC_CTYPE
+ * has to match to get sane results.
+ *
+ * To get predicable results on all platforms, we'll call the underlying
+ * routines with LC_ALL set to the appropriate locale for each set of members,
+ * and merge the results. Three members of the resulting object are therefore
+ * guaranteed to be encoded with LC_NUMERIC's codeset: "decimal_point",
+ * "thousands_sep" and "grouping". All other members are encoded with
+ * LC_MONETARY's codeset.
+ *
+ * Returns 0 on success. Returns non-zero on failure, and sets errno. On
+ * success, the caller is responsible for calling pg_localeconf_free() on the
+ * output struct to free the string members it contains.
+ */
+int
+pg_localeconv_r(const char *lc_monetary,
+ const char *lc_numeric,
+ struct lconv *output)
+{
+#ifdef WIN32
+ wchar_t *save_lc_ctype = NULL;
+ wchar_t *save_lc_monetary = NULL;
+ wchar_t *save_lc_numeric = NULL;
+ int save_config_thread_locale;
+ int result = -1;
+
+ /* Put setlocale() into thread-local mode. */
+ save_config_thread_locale = _configthreadlocale(_ENABLE_PER_THREAD_LOCALE);
+
+ /*
+ * Capture the current values as wide strings. Otherwise, we might not be
+ * able to restore them if their names contain non-ASCII characters and
+ * the intermediate locale changes the expected encoding. We don't want
+ * to leave the caller in an unexpected state by failing to restore, or
+ * crash the runtime library.
+ */
+ save_lc_ctype = _wsetlocale(LC_CTYPE, NULL);
+ if (!save_lc_ctype || !(save_lc_ctype = wcsdup(save_lc_ctype)))
+ goto exit;
+ save_lc_monetary = _wsetlocale(LC_MONETARY, NULL);
+ if (!save_lc_monetary || !(save_lc_monetary = wcsdup(save_lc_monetary)))
+ goto exit;
+ save_lc_numeric = _wsetlocale(LC_NUMERIC, NULL);
+ if (!save_lc_numeric || !(save_lc_numeric = wcsdup(save_lc_numeric)))
+ goto exit;
+
+ memset(output, 0, sizeof(*output));
+
+ /* Copy the LC_MONETARY members. */
+ if (!setlocale(LC_ALL, lc_monetary))
+ goto exit;
+ result = pg_localeconv_copy_members(output, localeconv(), LC_MONETARY);
+ if (result != 0)
+ goto exit;
+
+ /* Copy the LC_NUMERIC members. */
+ if (!setlocale(LC_ALL, lc_numeric))
+ goto exit;
+ result = pg_localeconv_copy_members(output, localeconv(), LC_NUMERIC);
+
+exit:
+ /* Restore everything we changed. */
+ if (save_lc_ctype)
+ {
+ _wsetlocale(LC_CTYPE, save_lc_ctype);
+ free(save_lc_ctype);
+ }
+ if (save_lc_monetary)
+ {
+ _wsetlocale(LC_MONETARY, save_lc_monetary);
+ free(save_lc_monetary);
+ }
+ if (save_lc_numeric)
+ {
+ _wsetlocale(LC_NUMERIC, save_lc_numeric);
+ free(save_lc_numeric);
+ }
+ _configthreadlocale(save_config_thread_locale);
+
+ return result;
+
+#else
+ locale_t monetary_locale;
+ locale_t numeric_locale;
+ int result;
+
+ /*
+ * All variations on Unix require locale_t objects for LC_MONETARY and
+ * LC_NUMERIC. We'll set all locale categories, so that we can don't have
+ * to worry about POSIX's undefined behavior if LC_CTYPE's encoding
+ * doesn't match.
+ */
+ errno = ENOENT;
+ monetary_locale = newlocale(LC_ALL_MASK, lc_monetary, 0);
+ if (monetary_locale == 0)
+ return -1;
+ numeric_locale = newlocale(LC_ALL_MASK, lc_numeric, 0);
+ if (numeric_locale == 0)
+ {
+ freelocale(monetary_locale);
+ return -1;
+ }
+
+ memset(output, 0, sizeof(*output));
+#if defined(TRANSLATE_FROM_LANGINFO)
+ /* Copy from non-standard nl_langinfo_l() extended items. */
+ result = pg_localeconv_from_langinfo(output,
+ monetary_locale,
+ numeric_locale);
+#elif defined(HAVE_LOCALE_CONV_L)
+ /* Copy the LC_MONETARY members from a thread-safe lconv object. */
+ result = pg_localeconv_copy_members(output,
+ localeconv_l(monetary_locale),
+ LC_MONETARY);
+ if (result != 0)
+ goto exit;
+ /* Copy the LC_NUMERIC members from a thread-safe lconv object. */
+ result = pg_localeconv_copy_members(output,
+ localeconv_l(numeric_locale),
+ LC_NUMERIC);
+#else
+ /* We have nothing better than standard POSIX facilities. */
+ {
+ static pthread_mutex_t big_lock = PTHREAD_MUTEX_INITIALIZER;
+ locale_t save_locale;
+
+ pthread_mutex_lock(&big_lock);
+ /* Copy the LC_MONETARY members. */
+ save_locale = uselocale(monetary_locale);
+ result = pg_localeconv_copy_members(output,
+ localeconv(),
+ LC_MONETARY);
+ if (result == 0)
+ {
+ /* Copy the LC_NUMERIC members. */
+ uselocale(numeric_locale);
+ result = pg_localeconv_copy_members(output,
+ localeconv(),
+ LC_NUMERIC);
+ }
+ pthread_mutex_unlock(&big_lock);
+
+ uselocale(save_locale);
+ }
+#endif
+
+ freelocale(monetary_locale);
+ freelocale(numeric_locale);
+
+ return result;
+#endif
+}
--
2.46.0
[text/x-patch] v4-0002-Use-thread-safe-strftime_l-instead-of-strftime.patch (7.9K, ../../CA+hUKGLqLxaNXN8mSqetYpD0QHWimHWZbi6KOfP=hfRvBfA9iw@mail.gmail.com/3-v4-0002-Use-thread-safe-strftime_l-instead-of-strftime.patch)
download | inline diff:
From 23c9b4455fdd170c7f0e390ab1b38849607836df Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Wed, 14 Aug 2024 23:06:59 +1200
Subject: [PATCH v4 2/3] Use thread-safe strftime_l() instead of strftime().
This removes some setlocale() calls and a lot of commentary about how
dangerous that is. strftime_l() is from POSIX 2008, and on Windows we
use _wcsftime_l().
While here, adjust error message for strftime_l() failure: it does not
set errno, so no %m.
Reviewed-by:
Discussion: https://postgr.es/m/CA%2BhUKGJqVe0%2BPv9dvC9dSums_PXxGo9SWcxYAMBguWJUGbWz-A%40mail.gmail.com
---
src/backend/main/main.c | 5 +-
src/backend/utils/adt/pg_locale.c | 113 ++++++++----------------------
2 files changed, 31 insertions(+), 87 deletions(-)
diff --git a/src/backend/main/main.c b/src/backend/main/main.c
index 4672aab8378..4ffe1fd596e 100644
--- a/src/backend/main/main.c
+++ b/src/backend/main/main.c
@@ -121,10 +121,7 @@ main(int argc, char *argv[])
init_locale("LC_MESSAGES", LC_MESSAGES, "");
#endif
- /*
- * We keep these set to "C" always, except transiently in pg_locale.c; see
- * that file for explanations.
- */
+ /* We keep these set to "C" always. See pg_locale.c for explanation. */
init_locale("LC_MONETARY", LC_MONETARY, "C");
init_locale("LC_NUMERIC", LC_NUMERIC, "C");
init_locale("LC_TIME", LC_TIME, "C");
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index dd4ba9e0e89..1cf8efcc7b7 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -18,34 +18,13 @@
* LC_MESSAGES is settable at run time and will take effect
* immediately.
*
- * The other categories, LC_MONETARY, LC_NUMERIC, and LC_TIME are also
- * settable at run-time. However, we don't actually set those locale
- * categories permanently. This would have bizarre effects like no
- * longer accepting standard floating-point literals in some locales.
- * Instead, we only set these locale categories briefly when needed,
- * cache the required information obtained from localeconv() or
- * strftime(), and then set the locale categories back to "C".
+ * The other categories, LC_MONETARY, LC_NUMERIC, and LC_TIME are
+ * permanentaly set to "C", and then we use temporary locale_t
+ * objects when we need to look up locale data based on the GUCs
+ * of the same name. Information is cached when the GUCs change.
* The cached information is only used by the formatting functions
* (to_char, etc.) and the money type. For the user, this should all be
* transparent.
- *
- * !!! NOW HEAR THIS !!!
- *
- * We've been bitten repeatedly by this bug, so let's try to keep it in
- * mind in future: on some platforms, the locale functions return pointers
- * to static data that will be overwritten by any later locale function.
- * Thus, for example, the obvious-looking sequence
- * save = setlocale(category, NULL);
- * if (!setlocale(category, value))
- * fail = true;
- * setlocale(category, save);
- * DOES NOT WORK RELIABLY: on some platforms the second setlocale() call
- * will change the memory save is pointing at. To do this sort of thing
- * safely, you *must* pstrdup what setlocale returns the first time.
- *
- * The POSIX locale standard is available here:
- *
- * http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap07.html
*----------
*/
@@ -663,8 +642,8 @@ PGLC_localeconv(void)
* pg_strftime(), which isn't locale-aware and does not need to be replaced.
*/
static size_t
-strftime_win32(char *dst, size_t dstlen,
- const char *format, const struct tm *tm)
+strftime_l_win32(char *dst, size_t dstlen,
+ const char *format, const struct tm *tm, locale_t locale)
{
size_t len;
wchar_t wformat[8]; /* formats used below need 3 chars */
@@ -680,7 +659,7 @@ strftime_win32(char *dst, size_t dstlen,
elog(ERROR, "could not convert format string from UTF-8: error code %lu",
GetLastError());
- len = wcsftime(wbuf, MAX_L10N_DATA, wformat, tm);
+ len = _wcsftime_l(wbuf, MAX_L10N_DATA, wformat, tm, locale);
if (len == 0)
{
/*
@@ -701,8 +680,8 @@ strftime_win32(char *dst, size_t dstlen,
return len;
}
-/* redefine strftime() */
-#define strftime(a,b,c,d) strftime_win32(a,b,c,d)
+/* redefine strftime_l() */
+#define strftime_l(a,b,c,d,e) strftime_l_win32(a,b,c,d,e)
#endif /* WIN32 */
/*
@@ -743,10 +722,7 @@ cache_locale_time(void)
bool strftimefail = false;
int encoding;
int i;
- char *save_lc_time;
-#ifdef WIN32
- char *save_lc_ctype;
-#endif
+ locale_t locale;
/* did we do this already? */
if (CurrentLCTimeValid)
@@ -754,39 +730,21 @@ cache_locale_time(void)
elog(DEBUG3, "cache_locale_time() executed; locale: \"%s\"", locale_time);
- /*
- * As in PGLC_localeconv(), it's critical that we not throw error while
- * libc's locale settings have nondefault values. Hence, we just call
- * strftime() within the critical section, and then convert and save its
- * results afterwards.
- */
-
- /* Save prevailing value of time locale */
- save_lc_time = setlocale(LC_TIME, NULL);
- if (!save_lc_time)
- elog(ERROR, "setlocale(NULL) failed");
- save_lc_time = pstrdup(save_lc_time);
-
+ errno = ENOENT;
#ifdef WIN32
-
- /*
- * On Windows, it appears that wcsftime() internally uses LC_CTYPE, so we
- * must set it here. This code looks the same as what PGLC_localeconv()
- * does, but the underlying reason is different: this does NOT determine
- * the encoding we'll get back from strftime_win32().
- */
-
- /* Save prevailing value of ctype locale */
- save_lc_ctype = setlocale(LC_CTYPE, NULL);
- if (!save_lc_ctype)
- elog(ERROR, "setlocale(NULL) failed");
- save_lc_ctype = pstrdup(save_lc_ctype);
-
- /* use lc_time to set the ctype */
- setlocale(LC_CTYPE, locale_time);
+ locale = _create_locale(LC_ALL, locale_time);
+ if (locale == (locale_t) 0)
+ _dosmaperr(GetLastError());
+#else
+ locale = newlocale(LC_ALL_MASK, locale_time, (locale_t) 0);
#endif
+ if (locale == (locale_t) 0)
+ {
+ if (errno == ENOMEM)
+ elog(ERROR, "out of memory");
- setlocale(LC_TIME, locale_time);
+ elog(ERROR, "coud not create locale \"%s\": %m", locale_time);
+ }
/* We use times close to current time as data for strftime(). */
timenow = time(NULL);
@@ -809,10 +767,10 @@ cache_locale_time(void)
for (i = 0; i < 7; i++)
{
timeinfo->tm_wday = i;
- if (strftime(bufptr, MAX_L10N_DATA, "%a", timeinfo) <= 0)
+ if (strftime_l(bufptr, MAX_L10N_DATA, "%a", timeinfo, locale) <= 0)
strftimefail = true;
bufptr += MAX_L10N_DATA;
- if (strftime(bufptr, MAX_L10N_DATA, "%A", timeinfo) <= 0)
+ if (strftime_l(bufptr, MAX_L10N_DATA, "%A", timeinfo, locale) <= 0)
strftimefail = true;
bufptr += MAX_L10N_DATA;
}
@@ -822,37 +780,26 @@ cache_locale_time(void)
{
timeinfo->tm_mon = i;
timeinfo->tm_mday = 1; /* make sure we don't have invalid date */
- if (strftime(bufptr, MAX_L10N_DATA, "%b", timeinfo) <= 0)
+ if (strftime_l(bufptr, MAX_L10N_DATA, "%b", timeinfo, locale) <= 0)
strftimefail = true;
bufptr += MAX_L10N_DATA;
- if (strftime(bufptr, MAX_L10N_DATA, "%B", timeinfo) <= 0)
+ if (strftime_l(bufptr, MAX_L10N_DATA, "%B", timeinfo, locale) <= 0)
strftimefail = true;
bufptr += MAX_L10N_DATA;
}
- /*
- * Restore the prevailing locale settings; as in PGLC_localeconv(),
- * failure to do so is fatal.
- */
#ifdef WIN32
- if (!setlocale(LC_CTYPE, save_lc_ctype))
- elog(FATAL, "failed to restore LC_CTYPE to \"%s\"", save_lc_ctype);
+ _free_locale(locale);
+#else
+ freelocale(locale);
#endif
- if (!setlocale(LC_TIME, save_lc_time))
- elog(FATAL, "failed to restore LC_TIME to \"%s\"", save_lc_time);
/*
* At this point we've done our best to clean up, and can throw errors, or
* call functions that might throw errors, with a clean conscience.
*/
if (strftimefail)
- elog(ERROR, "strftime() failed: %m");
-
- /* Release the pstrdup'd locale names */
- pfree(save_lc_time);
-#ifdef WIN32
- pfree(save_lc_ctype);
-#endif
+ elog(ERROR, "strftime_l() failed");
#ifndef WIN32
--
2.46.0
[text/x-patch] v4-0003-Remove-setlocale-from-check_locale.patch (10.7K, ../../CA+hUKGLqLxaNXN8mSqetYpD0QHWimHWZbi6KOfP=hfRvBfA9iw@mail.gmail.com/4-v4-0003-Remove-setlocale-from-check_locale.patch)
download | inline diff:
From 35e24316222ea444f9f8bb96656fecf102b7812a Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Thu, 15 Aug 2024 16:45:27 +1200
Subject: [PATCH v4 3/3] Remove setlocale() from check_locale().
Validate locale names with newlocale() or _create_locale() instead, to
avoid clobbering global state.
This also removes the "canonicalization" of the locale name, which
previously had two user-visible effects:
1. CREATE DATABASE ... LOCALE = '' would look up the locale from
environment variables, but this was not documented behavior and doesn't
seem too useful. A default will normally be inherited from the template
if you just leave the option off. (Note that initdb still chooses
default values from the server environment, and that would often be the
original source of the template database's values.)
2. On Windows only, the setlocale() step reached by CREATE DATABASE ...
LOCALE = '...' did accept a wide range of formats and do some
canonicalization, when using (deprecated) pre-BCP 47 locale names, but
again it seems enough to let that happen in the initdb phase only.
Now, CREATE DATABASE and SET lc_XXX reject ''. CREATE COLLATION
continues to accept it, as it never recorded canonicalized values so
there may be system catalogs containing verbatim '' in the wild. We'll
need to research the upgrade implications before banning it there.
Thanks to Peter Eisentraut and Tom Lane for the suggestion that we might
not really need to preserve those behaviors in the backend, in our hunt
for non-thread-safe code.
Reviewed-by: Heikki Linnakangas <[email protected]>
Discussion: https://postgr.es/m/CA%2BhUKGJqVe0%2BPv9dvC9dSums_PXxGo9SWcxYAMBguWJUGbWz-A%40mail.gmail.com
Discussion: https://postgr.es/m/CA%2BhUKGK57sgUYKO03jB4VarTsswfMyScFAyJpVnYD8c%2Bg12_mg%40mail.gmail.com
---
src/backend/commands/dbcommands.c | 7 +-
src/backend/utils/adt/pg_locale.c | 167 ++++++++++++++++++------------
src/bin/initdb/initdb.c | 2 -
src/include/port/win32_port.h | 1 -
src/include/utils/pg_locale.h | 2 +-
5 files changed, 104 insertions(+), 75 deletions(-)
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index 7c92c3463b6..eaf072fff90 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -728,7 +728,6 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
const char *dblocale = NULL;
char *dbicurules = NULL;
char dblocprovider = '\0';
- char *canonname;
int encoding = -1;
bool dbistemplate = false;
bool dballowconnections = true;
@@ -1062,18 +1061,16 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
errmsg("invalid server encoding %d", encoding)));
/* Check that the chosen locales are valid, and get canonical spellings */
- if (!check_locale(LC_COLLATE, dbcollate, &canonname))
+ if (!check_locale(LC_COLLATE, dbcollate))
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
errmsg("invalid LC_COLLATE locale name: \"%s\"", dbcollate),
errhint("If the locale name is specific to ICU, use ICU_LOCALE.")));
- dbcollate = canonname;
- if (!check_locale(LC_CTYPE, dbctype, &canonname))
+ if (!check_locale(LC_CTYPE, dbctype))
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
errmsg("invalid LC_CTYPE locale name: \"%s\"", dbctype),
errhint("If the locale name is specific to ICU, use ICU_LOCALE.")));
- dbctype = canonname;
check_encoding_locale_matches(encoding, dbcollate, dbctype);
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 1cf8efcc7b7..383ae21dab0 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -190,6 +190,65 @@ wcstombs_l(char *dest, const wchar_t *src, size_t n, locale_t loc)
}
#endif
+/*
+ * The category names as strings. These are the names of the environment
+ * variables that define the server locale environment. We always unset
+ * LC_ALL, so we only need the actual categories.
+ */
+static const char *
+get_lc_category_name(int category)
+{
+ switch (category)
+ {
+ case LC_COLLATE:
+ return "LC_COLLATE";
+ case LC_CTYPE:
+ return "LC_CTYPE";
+#ifdef LC_MESSAGES
+ case LC_MESSAGES:
+ return "LC_MESSAGES";
+#endif
+ case LC_MONETARY:
+ return "LC_MONETARY";
+ case LC_NUMERIC:
+ return "LC_NUMERIC";
+ case LC_TIME:
+ return "LC_TIME";
+ default:
+ return NULL;
+ };
+};
+
+#ifndef WIN32
+/*
+ * The newlocale() function needs LC_xxx_MASK, but sometimes we have LC_xxx,
+ * and POSIX doesn't offer a way to translate.
+ */
+static int
+get_lc_category_mask(int category)
+{
+ switch (category)
+ {
+ case LC_COLLATE:
+ return LC_COLLATE_MASK;
+ case LC_CTYPE:
+ return LC_CTYPE_MASK;
+#ifdef LC_MESSAGES
+ case LC_MESSAGES:
+ return LC_MESSAGES_MASK;
+#endif
+ case LC_MONETARY:
+ return LC_MONETARY_MASK;
+ case LC_NUMERIC:
+ return LC_NUMERIC_MASK;
+ case LC_TIME:
+ return LC_TIME_MASK;
+ default:
+ return 0;
+ };
+}
+#endif
+
/*
* pg_perm_setlocale
*
@@ -257,38 +316,9 @@ pg_perm_setlocale(int category, const char *locale)
#endif
}
- switch (category)
- {
- case LC_COLLATE:
- envvar = "LC_COLLATE";
- break;
- case LC_CTYPE:
- envvar = "LC_CTYPE";
- break;
-#ifdef LC_MESSAGES
- case LC_MESSAGES:
- envvar = "LC_MESSAGES";
-#ifdef WIN32
- result = IsoLocaleName(locale);
- if (result == NULL)
- result = (char *) locale;
- elog(DEBUG3, "IsoLocaleName() executed; locale: \"%s\"", result);
-#endif /* WIN32 */
- break;
-#endif /* LC_MESSAGES */
- case LC_MONETARY:
- envvar = "LC_MONETARY";
- break;
- case LC_NUMERIC:
- envvar = "LC_NUMERIC";
- break;
- case LC_TIME:
- envvar = "LC_TIME";
- break;
- default:
- elog(FATAL, "unrecognized LC category: %d", category);
- return NULL; /* keep compiler quiet */
- }
+ envvar = get_lc_category_name(category);
+ if (!envvar)
+ elog(FATAL, "unrecognized LC category: %d", category);
if (setenv(envvar, result, 1) != 0)
return NULL;
@@ -299,43 +329,48 @@ pg_perm_setlocale(int category, const char *locale)
/*
* Is the locale name valid for the locale category?
- *
- * If successful, and canonname isn't NULL, a palloc'd copy of the locale's
- * canonical name is stored there. This is especially useful for figuring out
- * what locale name "" means (ie, the server environment value). (Actually,
- * it seems that on most implementations that's the only thing it's good for;
- * we could wish that setlocale gave back a canonically spelled version of
- * the locale name, but typically it doesn't.)
*/
bool
-check_locale(int category, const char *locale, char **canonname)
+check_locale(int category, const char *locale)
{
- char *save;
- char *res;
-
- if (canonname)
- *canonname = NULL; /* in case of failure */
-
- save = setlocale(category, NULL);
- if (!save)
- return false; /* won't happen, we hope */
+ locale_t loc;
- /* save may be pointing at a modifiable scratch variable, see above. */
- save = pstrdup(save);
-
- /* set the locale with setlocale, to see if it accepts it. */
- res = setlocale(category, locale);
+ if (locale[0] == 0)
+ {
+ /*
+ * We only accept explicit locale names, not "". We don't want to
+ * rely on environment variables in the backend.
+ */
+ return false;
+ }
- /* save canonical name if requested. */
- if (res && canonname)
- *canonname = pstrdup(res);
+ /*
+ * See if we can open it. Unfortunately we can't always distinguish
+ * out-of-memory from invalid locale name.
+ */
+ errno = ENOENT;
+#ifdef WIN32
+ loc = _create_locale(category, locale);
+ if (loc == (locale_t) 0)
+ _dosmaperr(GetLastError());
+#else
+ loc = newlocale(get_lc_category_mask(category), locale, (locale_t) 0);
+#endif
+ if (loc == (locale_t) 0)
+ {
+ if (errno == ENOMEM)
+ elog(ERROR, "out of memory");
- /* restore old value. */
- if (!setlocale(category, save))
- elog(WARNING, "failed to restore old locale \"%s\"", save);
- pfree(save);
+ /* Otherwise assume the locale doesn't exist. */
+ return false;
+ }
+#ifdef WIN32
+ _free_locale(loc);
+#else
+ freelocale(loc);
+#endif
- return (res != NULL);
+ return true;
}
@@ -353,7 +388,7 @@ check_locale(int category, const char *locale, char **canonname)
bool
check_locale_monetary(char **newval, void **extra, GucSource source)
{
- return check_locale(LC_MONETARY, *newval, NULL);
+ return check_locale(LC_MONETARY, *newval);
}
void
@@ -365,7 +400,7 @@ assign_locale_monetary(const char *newval, void *extra)
bool
check_locale_numeric(char **newval, void **extra, GucSource source)
{
- return check_locale(LC_NUMERIC, *newval, NULL);
+ return check_locale(LC_NUMERIC, *newval);
}
void
@@ -377,7 +412,7 @@ assign_locale_numeric(const char *newval, void *extra)
bool
check_locale_time(char **newval, void **extra, GucSource source)
{
- return check_locale(LC_TIME, *newval, NULL);
+ return check_locale(LC_TIME, *newval);
}
void
@@ -413,7 +448,7 @@ check_locale_messages(char **newval, void **extra, GucSource source)
* On Windows, we can't even check the value, so accept blindly
*/
#if defined(LC_MESSAGES) && !defined(WIN32)
- return check_locale(LC_MESSAGES, *newval, NULL);
+ return check_locale(LC_MESSAGES, *newval);
#else
return true;
#endif
diff --git a/src/bin/initdb/initdb.c b/src/bin/initdb/initdb.c
index f00718a0150..1aa1c26785c 100644
--- a/src/bin/initdb/initdb.c
+++ b/src/bin/initdb/initdb.c
@@ -2126,8 +2126,6 @@ locale_date_order(const char *locale)
* it seems that on most implementations that's the only thing it's good for;
* we could wish that setlocale gave back a canonically spelled version of
* the locale name, but typically it doesn't.)
- *
- * this should match the backend's check_locale() function
*/
static void
check_locale_name(int category, const char *locale, char **canonname)
diff --git a/src/include/port/win32_port.h b/src/include/port/win32_port.h
index 7ffe5891c69..f2c92525b61 100644
--- a/src/include/port/win32_port.h
+++ b/src/include/port/win32_port.h
@@ -471,7 +471,6 @@ extern char *pgwin32_setlocale(int category, const char *locale);
#define setlocale(a,b) pgwin32_setlocale(a,b)
-
/* In backend/port/win32/signal.c */
extern PGDLLIMPORT volatile int pg_signal_queue;
extern PGDLLIMPORT int pg_signal_mask;
diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h
index f41d33975be..5b73526c92c 100644
--- a/src/include/utils/pg_locale.h
+++ b/src/include/utils/pg_locale.h
@@ -51,7 +51,7 @@ extern PGDLLIMPORT char *localized_full_months[];
/* is the databases's LC_CTYPE the C locale? */
extern PGDLLIMPORT bool database_ctype_is_c;
-extern bool check_locale(int category, const char *locale, char **canonname);
+extern bool check_locale(int category, const char *locale);
extern char *pg_perm_setlocale(int category, const char *locale);
extern bool lc_collate_is_c(Oid collation);
--
2.46.0
^ permalink raw reply [nested|flat] 51+ messages in thread
* Re: Thread-safe nl_langinfo() and localeconv()
@ 2024-08-28 19:07 Peter Eisentraut <[email protected]>
parent: Thomas Munro <[email protected]>
0 siblings, 0 replies; 51+ messages in thread
From: Peter Eisentraut @ 2024-08-28 19:07 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; Heikki Linnakangas <[email protected]>; +Cc: pgsql-hackers
On 16.08.24 02:48, Thomas Munro wrote:
> 2. A similar argument applies to Windows canonicalisation. CREATE
> COLLATION isn't doing it. CREATE DATABASE is, but again, what is the
> point? See previous.
I don't really know about Windows locales. But we are doing
canonicalization of ICU locale names now. So there seems to be a desire
to do canonicalization in general. (Obviously, if we're doing it
poorly, then we don't have to keep it that way indefinitely.)
^ permalink raw reply [nested|flat] 51+ messages in thread
end of thread, other threads:[~2024-08-28 19:07 UTC | newest]
Thread overview: 51+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-05-01 15:52 Re: Cached plans and statement generalization Robert Haas <[email protected]>
2017-05-02 09:50 ` Konstantin Knizhnik <[email protected]>
2017-05-02 18:26 ` Robert Haas <[email protected]>
2017-05-10 16:11 ` Konstantin Knizhnik <[email protected]>
2017-05-11 15:12 ` Bruce Momjian <[email protected]>
2017-05-11 17:39 ` Douglas Doole <[email protected]>
2017-05-11 18:15 ` Bruce Momjian <[email protected]>
2017-05-11 18:31 ` Tom Lane <[email protected]>
2017-05-11 18:32 ` Andres Freund <[email protected]>
2017-05-11 19:48 ` Konstantin Knizhnik <[email protected]>
2017-05-11 19:52 ` Andres Freund <[email protected]>
2017-05-11 20:30 ` Konstantin Knizhnik <[email protected]>
2017-05-11 19:41 ` Konstantin Knizhnik <[email protected]>
2017-05-12 00:58 ` Bruce Momjian <[email protected]>
2017-05-12 07:50 ` Konstantin Knizhnik <[email protected]>
2017-05-12 15:23 ` Bruce Momjian <[email protected]>
2017-05-12 17:35 ` Konstantin Knizhnik <[email protected]>
2017-05-12 17:43 ` Bruce Momjian <[email protected]>
2017-05-12 17:38 ` Andres Freund <[email protected]>
2017-05-15 15:31 ` Robert Haas <[email protected]>
2017-05-18 08:57 ` Konstantin Knizhnik <[email protected]>
2017-05-18 16:47 ` Andres Freund <[email protected]>
2017-05-25 15:54 ` Konstantin Knizhnik <[email protected]>
2017-09-09 03:35 ` Thomas Munro <[email protected]>
2017-09-11 09:24 ` Konstantin Knizhnik <[email protected]>
2017-09-12 17:11 ` Konstantin Knizhnik <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2021-01-13 11:53 [PATCH v3 2/2] Invalidate relation map cache in subscriber syscache invalidation callbacks Bharath Rupireddy <[email protected]>
2024-08-15 08:03 Re: Thread-safe nl_langinfo() and localeconv() Thomas Munro <[email protected]>
2024-08-15 11:11 ` Re: Thread-safe nl_langinfo() and localeconv() Heikki Linnakangas <[email protected]>
2024-08-16 00:48 ` Re: Thread-safe nl_langinfo() and localeconv() Thomas Munro <[email protected]>
2024-08-28 19:07 ` Re: Thread-safe nl_langinfo() and localeconv() Peter Eisentraut <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox