public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. 4+ messages / 3 participants [nested] [flat]
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. @ 2020-12-03 07:54 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 4+ messages in thread From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw) OUTDATED is added a new unreserved keyword. When used, REINDEX will only process indexes that have an outdated dependency. For now, only dependency on collations are supported but we'll likely support other kind of dependency in the future. Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so client code can filter such indexes if needed. This function will also be used in a following commit to teach reindexdb to use this new OUTDATED option and order the tables by the amount of work that will actually be done. Catversion (should be) bumped. Author: Julien Rouhaud <[email protected]> Reviewed-by: Discussion: https://postgr.es/m/20201203093143.GA64934%40nol --- doc/src/sgml/func.sgml | 27 ++++-- doc/src/sgml/ref/reindex.sgml | 12 +++ src/backend/catalog/index.c | 107 ++++++++++++++++++++- src/backend/commands/indexcmds.c | 12 ++- src/backend/parser/gram.y | 4 +- src/backend/utils/cache/relcache.c | 40 ++++++++ src/bin/psql/tab-complete.c | 2 +- src/include/catalog/index.h | 3 + src/include/catalog/pg_proc.dat | 4 + src/include/parser/kwlist.h | 1 + src/include/utils/relcache.h | 1 + src/test/regress/expected/create_index.out | 10 ++ src/test/regress/sql/create_index.sql | 10 ++ 13 files changed, 221 insertions(+), 12 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index bf99f82149..2cf6e66234 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size <para> <xref linkend="functions-admin-index-table"/> shows the functions - available for index maintenance tasks. (Note that these maintenance - tasks are normally done automatically by autovacuum; use of these - functions is only required in special cases.) - These functions cannot be executed during recovery. - Use of these functions is restricted to superusers and the owner - of the given index. + available for index maintenance tasks. (Note that the maintenance + tasks performing actions on indexes are normally done automatically by + autovacuum; use of these functions is only required in special cases.) + The functions performing actions on indexes cannot be executed during + recovery. + Use of the functions performing actions on indexes is restricted to + superusers and the owner of the given index. </para> <table id="functions-admin-index-table"> @@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size option. </para></entry> </row> + + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>pg_index_has_outdated_dependency</primary> + </indexterm> + <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> ) + <returnvalue>boolean</returnvalue> + </para> + <para> + Check if the specified index has any outdated dependency. For now only + dependency on collations are supported. + </para></entry> + </row> </tbody> </tgroup> </table> diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index b22d39eba9..2d94d49cde 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN <phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase> CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ] + OUTDATED [ <replaceable class="parameter">boolean</replaceable> ] TABLESPACE <replaceable class="parameter">new_tablespace</replaceable> VERBOSE [ <replaceable class="parameter">boolean</replaceable> ] </synopsis> @@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN </listitem> </varlistentry> + <varlistentry> + <term><literal>OUTDATED</literal></term> + <listitem> + <para> + This option can be used to filter the list of indexes to rebuild and only + process indexes that have outdated dependencies. Fow now, the only + handle dependency is for the collation provider version. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>TABLESPACE</literal></term> <listitem> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..571feac5db 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -100,6 +100,12 @@ typedef struct Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER]; } SerializedReindexState; +typedef struct +{ + Oid relid; /* targetr index oid */ + bool outdated; /* depends on at least on deprected collation? */ +} IndexHasOutdatedColl; + /* non-export function prototypes */ static bool relationHasPrimaryKey(Relation rel); static TupleDesc ConstructTupleDescriptor(Relation heapRelation, @@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid) list_free(context.warned_colls); } +/* + * Detect if an index depends on at least one outdated collation. + * This is a callback for visitDependenciesOf(). + */ +static bool +do_check_index_has_outdated_collation(const ObjectAddress *otherObject, + const char *version, + char **new_version, + void *data) +{ + IndexHasOutdatedColl *context = data; + char *current_version; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return false; + + /* Fast exit if we already found a outdated collation version. */ + if (context->outdated) + return false; + + /* Ask the provider for the current version. Give up if unsupported. */ + current_version = get_collation_version_for_oid(otherObject->objectId, + false); + if (!current_version) + return false; + + if (!version || strcmp(version, current_version) != 0) + context->outdated = true; + + return false; +} + +Datum +pg_index_has_outdated_dependency(PG_FUNCTION_ARGS) +{ + Oid indexOid = PG_GETARG_OID(0); + Relation rel; + bool isIndex; + bool res; + + rel = try_relation_open(indexOid, AccessShareLock); + + if (rel == NULL) + PG_RETURN_NULL(); + + isIndex = rel->rd_rel->relkind == RELKIND_INDEX; + + if (!isIndex) + { + relation_close(rel, AccessShareLock); + PG_RETURN_NULL(); + } + + res = index_has_outdated_dependency(indexOid); + + relation_close(rel, AccessShareLock); + + PG_RETURN_BOOL(res); +} + +/* + * Check whether the given index has a dependency with an outdated + * collation version. + * Caller must hold a suitable lock and make sure that the given Oid belongs to + * an index. + */ +bool +index_has_outdated_collation(Oid indexOid) +{ + ObjectAddress object; + IndexHasOutdatedColl context; + + object.classId = RelationRelationId; + object.objectId = indexOid; + object.objectSubId = 0; + + context.relid = indexOid; + context.outdated = false; + + visitDependenciesOf(&object, &do_check_index_has_outdated_collation, + &context); + + return context.outdated; +} + +/* + * Check whether the given index has a dependency with an outdated + * refobjversion. + * Caller must hold a suitable lock and make sure that the given Oid belongs to + * an index. + * For now, only dependency on collations are supported. + */ +bool +index_has_outdated_dependency(Oid indexOid) +{ + return index_has_outdated_collation(indexOid); +} + /* * Update the version for collations. A callback for visitDependenciesOf(). */ @@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params) * relcache to get this with a sequential scan if ignoring system * indexes.) */ - indexIds = RelationGetIndexList(rel); + indexIds = RelationGetIndexListFiltered(rel, params->options); if (flags & REINDEX_REL_SUPPRESS_INDEX_USE) { diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 8bc652ecd3..0e1bcad101 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel) bool concurrently = false; bool verbose = false; char *tablespacename = NULL; + bool outdated_filter = false; /* Parse option list */ foreach(lc, stmt->params) @@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel) concurrently = defGetBoolean(opt); else if (strcmp(opt->defname, "tablespace") == 0) tablespacename = defGetString(opt); + else if (strcmp(opt->defname, "outdated") == 0) + outdated_filter = defGetBoolean(opt); else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel) params.options = (verbose ? REINDEXOPT_VERBOSE : 0) | - (concurrently ? REINDEXOPT_CONCURRENTLY : 0); + (concurrently ? REINDEXOPT_CONCURRENTLY : 0) | + (outdated_filter ? REINDEXOPT_OUTDATED : 0); /* * Assign the tablespace OID to move indexes to, with InvalidOid to do @@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params) RelationGetRelationName(heapRelation)))); /* Add all the valid indexes of relation to list */ - foreach(lc, RelationGetIndexList(heapRelation)) + foreach(lc, RelationGetIndexListFiltered(heapRelation, + params->options)) { Oid cellOid = lfirst_oid(lc); Relation indexRelation = index_open(cellOid, @@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params) MemoryContextSwitchTo(oldcontext); - foreach(lc2, RelationGetIndexList(toastRelation)) + foreach(lc2, RelationGetIndexListFiltered(toastRelation, + params->options)) { Oid cellOid = lfirst_oid(lc2); Relation indexRelation = index_open(cellOid, diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 652be0b96d..a67cfa867c 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); NULLS_P NUMERIC OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR - ORDER ORDINALITY OTHERS OUT_P OUTER_P + ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY @@ -15430,6 +15430,7 @@ unreserved_keyword: | OPTIONS | ORDINALITY | OTHERS + | OUTDATED | OVER | OVERRIDING | OWNED @@ -16002,6 +16003,7 @@ bare_label_keyword: | ORDINALITY | OTHERS | OUT_P + | OUTDATED | OUTER_P | OVERLAY | OVERRIDING diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index 7ef510cd01..06545ee550 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation) return result; } +/* + * RelationGetIndexListFiltered -- get a filtered list of indexes on this + * relation. + * + * Calls RelationGetIndexList and only keep indexes that have an outdated + * dependency. For now, only collation version dependency is supported. + */ +List * +RelationGetIndexListFiltered(Relation relation, bits32 options) +{ + List *result, + *full_list; + ListCell *lc; + + full_list = RelationGetIndexList(relation); + + /* Fast exit if no filtering was asked, or if the list if empty. */ + if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL) + return full_list; + + result = NIL; + foreach(lc, full_list) + { + Oid indexOid = lfirst_oid(lc); + + /* + * Check for outdated collation version dependency. + */ + if (index_has_outdated_collation(indexOid)) + { + result = lappend_oid(result, indexOid); + continue; + } + + /* Didn't find any outdated dependency, index will be ignored. */ + } + + return result; +} + /* * RelationGetStatExtList * get a list of OIDs of statistics objects on this relation diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index 9f0208ac49..fac3e0476a 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end) * one word, so the above test is correct. */ if (ends_with(prev_wd, '(') || ends_with(prev_wd, ',')) - COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE"); + COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE"); else if (TailMatches("TABLESPACE")) COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces); } diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index e22d506436..298c0c633c 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -42,6 +42,7 @@ typedef struct ReindexParams #define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */ #define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */ #define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */ +#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */ /* state info for validate_index bulkdelete callback */ typedef struct ValidateIndexState @@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo, bool *isnull); extern void index_check_collation_versions(Oid relid); +extern bool index_has_outdated_collation(Oid indexOid); +extern bool index_has_outdated_dependency(Oid indexOid); extern void index_update_collation_versions(Oid relid, Oid coll); extern void index_build(Relation heapRelation, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 3d3974f467..4874d33996 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -949,6 +949,10 @@ proname => 'pg_indexam_has_property', provolatile => 's', prorettype => 'bool', proargtypes => 'oid text', prosrc => 'pg_indexam_has_property' }, +{ oid => '8102', descr => 'test property of an index', + proname => 'pg_index_has_outdated_dependency', provolatile => 's', + prorettype => 'bool', proargtypes => 'regclass', + prosrc => 'pg_index_has_outdated_dependency' }, { oid => '637', descr => 'test property of an index', proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool', proargtypes => 'regclass text', prosrc => 'pg_index_has_property' }, diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..e6c725d9a6 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL) PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL) +PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL) PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL) diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h index 2fcdf79323..a7a2272abd 100644 --- a/src/include/utils/relcache.h +++ b/src/include/utils/relcache.h @@ -45,6 +45,7 @@ extern void RelationClose(Relation relation); */ extern List *RelationGetFKeyList(Relation relation); extern List *RelationGetIndexList(Relation relation); +extern List *RelationGetIndexListFiltered(Relation relation, bits32 options); extern List *RelationGetStatExtList(Relation relation); extern Oid RelationGetPrimaryKeyIndex(Relation relation); extern Oid RelationGetReplicaIndex(Relation relation); diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index 830fdddf24..5f7c49c650 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed \set VERBOSITY default DROP TABLE reindex_verbose; -- +-- REINDEX (OUTDATED) +-- +CREATE TABLE reindex_coll(id integer primary key); +\set VERBOSITY terse \\ -- suppress machine-dependent details +-- no suitable index should be found +REINDEX (OUTDATED) TABLE reindex_coll; +NOTICE: table "reindex_coll" has no indexes to reindex +\set VERBOSITY default +DROP TABLE reindex_coll; +-- -- REINDEX CONCURRENTLY -- CREATE TABLE concur_reindex_tab (c1 int); diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index 8bc76f7c6f..808cacee5d 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose; \set VERBOSITY default DROP TABLE reindex_verbose; +-- +-- REINDEX (OUTDATED) +-- +CREATE TABLE reindex_coll(id integer primary key); +\set VERBOSITY terse \\ -- suppress machine-dependent details +-- no suitable index should be found +REINDEX (OUTDATED) TABLE reindex_coll; +\set VERBOSITY default +DROP TABLE reindex_coll; + -- -- REINDEX CONCURRENTLY -- -- 2.30.1 --oao5se3im4osqowp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch" ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: [PATCH] plpython function causes server panic @ 2023-12-02 01:04 Tom Lane <[email protected]> 0 siblings, 1 reply; 4+ messages in thread From: Tom Lane @ 2023-12-02 01:04 UTC (permalink / raw) To: Hao Zhang <[email protected]>; +Cc: [email protected] Hao Zhang <[email protected]> writes: > I found a problem when executing the plpython function: > After the plpython function returns an error, in the same session, if we > continue to execute > plpython function, the server panic will be caused. Thanks for the report! I see the problem is that we're not expecting BeginInternalSubTransaction to fail. However, I'm not sure I like this solution, mainly because it's only covering a fraction of the problem. There are similarly unsafe usages in plperl, pltcl, and very possibly a lot of third-party PLs. I wonder if there's a way to deal with this issue without changing these API assumptions. The only readily-reachable error case in BeginInternalSubTransaction is this specific one about IsInParallelMode, which was added later than the original design and evidently with not a lot of thought or testing. The comment for it speculates about whether we could get rid of it, so I wonder if our thoughts about this ought to go in that direction. In any case, if we do proceed along the lines of catching errors from BeginInternalSubTransaction, I think your patch is a bit shy of a load because it doesn't do all the same things that other callers of PLy_spi_exception_set do. Looking at that made me wonder why the PLy_spi_exceptions lookup business was being duplicated by every caller rather than being done once in PLy_spi_exception_set. So 0001 attached is a quick refactoring patch to remove that code duplication, and then 0002 is your patch adapted to that. I also attempted to include a test case in 0002, but I'm not very satisfied with that. Your original test case seemed pretty expensive for the amount of code coverage it adds, so I tried to make it happen with debug_parallel_query instead. That does exercise the new code, but it does not exhibit the crash if run against unpatched code. That's because with this test case the error is only thrown in worker processes not the leader, so we don't end up with corrupted Python state in the leader. That result also points up that the original test case isn't very reliable for this either: you have to have parallel_leader_participation on, and you have to have the leader process at least one row, which makes it pretty timing-sensitive. On top of all that, the test would become useless if we do eventually get rid of the !IsInParallelMode restriction. So I'm kind of inclined to not bother with a test case if this gets to be committed in this form. Thoughts anyone? regards, tom lane Attachments: [text/x-diff] v2-0001-simplify-PLy_spi_exception_set-API.patch (3.6K, ../../[email protected]/2-v2-0001-simplify-PLy_spi_exception_set-API.patch) download | inline diff: diff --git a/src/pl/plpython/plpy_spi.c b/src/pl/plpython/plpy_spi.c index ff87b27de0..64ca47f218 100644 --- a/src/pl/plpython/plpy_spi.c +++ b/src/pl/plpython/plpy_spi.c @@ -28,7 +28,7 @@ static PyObject *PLy_spi_execute_query(char *query, long limit); static PyObject *PLy_spi_execute_fetch_result(SPITupleTable *tuptable, uint64 rows, int status); -static void PLy_spi_exception_set(PyObject *excclass, ErrorData *edata); +static void PLy_spi_exception_set(ErrorData *edata); /* prepare(query="select * from foo") @@ -470,8 +470,6 @@ PLy_commit(PyObject *self, PyObject *args) PG_CATCH(); { ErrorData *edata; - PLyExceptionEntry *entry; - PyObject *exc; /* Save error info */ MemoryContextSwitchTo(oldcontext); @@ -481,17 +479,8 @@ PLy_commit(PyObject *self, PyObject *args) /* was cleared at transaction end, reset pointer */ exec_ctx->scratch_ctx = NULL; - /* Look up the correct exception */ - entry = hash_search(PLy_spi_exceptions, &(edata->sqlerrcode), - HASH_FIND, NULL); - - /* - * This could be a custom error code, if that's the case fallback to - * SPIError - */ - exc = entry ? entry->exc : PLy_exc_spi_error; /* Make Python raise the exception */ - PLy_spi_exception_set(exc, edata); + PLy_spi_exception_set(edata); FreeErrorData(edata); return NULL; @@ -517,8 +506,6 @@ PLy_rollback(PyObject *self, PyObject *args) PG_CATCH(); { ErrorData *edata; - PLyExceptionEntry *entry; - PyObject *exc; /* Save error info */ MemoryContextSwitchTo(oldcontext); @@ -528,17 +515,8 @@ PLy_rollback(PyObject *self, PyObject *args) /* was cleared at transaction end, reset pointer */ exec_ctx->scratch_ctx = NULL; - /* Look up the correct exception */ - entry = hash_search(PLy_spi_exceptions, &(edata->sqlerrcode), - HASH_FIND, NULL); - - /* - * This could be a custom error code, if that's the case fallback to - * SPIError - */ - exc = entry ? entry->exc : PLy_exc_spi_error; /* Make Python raise the exception */ - PLy_spi_exception_set(exc, edata); + PLy_spi_exception_set(edata); FreeErrorData(edata); return NULL; @@ -594,8 +572,6 @@ void PLy_spi_subtransaction_abort(MemoryContext oldcontext, ResourceOwner oldowner) { ErrorData *edata; - PLyExceptionEntry *entry; - PyObject *exc; /* Save error info */ MemoryContextSwitchTo(oldcontext); @@ -607,17 +583,8 @@ PLy_spi_subtransaction_abort(MemoryContext oldcontext, ResourceOwner oldowner) MemoryContextSwitchTo(oldcontext); CurrentResourceOwner = oldowner; - /* Look up the correct exception */ - entry = hash_search(PLy_spi_exceptions, &(edata->sqlerrcode), - HASH_FIND, NULL); - - /* - * This could be a custom error code, if that's the case fallback to - * SPIError - */ - exc = entry ? entry->exc : PLy_exc_spi_error; /* Make Python raise the exception */ - PLy_spi_exception_set(exc, edata); + PLy_spi_exception_set(edata); FreeErrorData(edata); } @@ -626,12 +593,21 @@ PLy_spi_subtransaction_abort(MemoryContext oldcontext, ResourceOwner oldowner) * internal query and error position. */ static void -PLy_spi_exception_set(PyObject *excclass, ErrorData *edata) +PLy_spi_exception_set(ErrorData *edata) { + PLyExceptionEntry *entry; + PyObject *excclass; PyObject *args = NULL; PyObject *spierror = NULL; PyObject *spidata = NULL; + /* Look up the correct exception */ + entry = hash_search(PLy_spi_exceptions, &(edata->sqlerrcode), + HASH_FIND, NULL); + + /* Fall back to SPIError if no entry */ + excclass = entry ? entry->exc : PLy_exc_spi_error; + args = Py_BuildValue("(s)", edata->message); if (!args) goto failure; [text/x-diff] v2-0002-fix-plpython-subtrans-start-failure.patch (6.8K, ../../[email protected]/3-v2-0002-fix-plpython-subtrans-start-failure.patch) download | inline diff: diff --git a/src/pl/plpython/expected/plpython_spi.out b/src/pl/plpython/expected/plpython_spi.out index 8853e2540d..26e1cc3c9a 100644 --- a/src/pl/plpython/expected/plpython_spi.out +++ b/src/pl/plpython/expected/plpython_spi.out @@ -464,3 +464,29 @@ SELECT plan_composite_args(); (3,label) (1 row) +-- +-- Error cases +-- +-- Check recovery from a subtransaction start failure; to make one +-- happen repeatably, try to do a SPI operation in an allegedly +-- parallel-safe function +CREATE FUNCTION falsely_marked_parallel_safe() RETURNS SETOF int AS +$$ +plpy.execute("select 2+2") +for i in range(0, 5): + yield (i) +$$ LANGUAGE plpython3u parallel safe; +SET debug_parallel_query = 1; +SELECT falsely_marked_parallel_safe(); +ERROR: error fetching next item from iterator +DETAIL: spiexceptions.InvalidTransactionState: cannot start subtransactions during a parallel operation +CONTEXT: Traceback (most recent call last): +PL/Python function "falsely_marked_parallel_safe" +parallel worker +SELECT falsely_marked_parallel_safe(); +ERROR: error fetching next item from iterator +DETAIL: spiexceptions.InvalidTransactionState: cannot start subtransactions during a parallel operation +CONTEXT: Traceback (most recent call last): +PL/Python function "falsely_marked_parallel_safe" +parallel worker +RESET debug_parallel_query; diff --git a/src/pl/plpython/plpy_cursorobject.c b/src/pl/plpython/plpy_cursorobject.c index 57e8f8ec21..8bfe05815b 100644 --- a/src/pl/plpython/plpy_cursorobject.c +++ b/src/pl/plpython/plpy_cursorobject.c @@ -98,7 +98,8 @@ PLy_cursor_query(const char *query) oldcontext = CurrentMemoryContext; oldowner = CurrentResourceOwner; - PLy_spi_subtransaction_begin(oldcontext, oldowner); + if (!PLy_spi_subtransaction_begin(oldcontext, oldowner)) + return NULL; PG_TRY(); { @@ -196,7 +197,8 @@ PLy_cursor_plan(PyObject *ob, PyObject *args) oldcontext = CurrentMemoryContext; oldowner = CurrentResourceOwner; - PLy_spi_subtransaction_begin(oldcontext, oldowner); + if (!PLy_spi_subtransaction_begin(oldcontext, oldowner)) + return NULL; PG_TRY(); { @@ -333,7 +335,8 @@ PLy_cursor_iternext(PyObject *self) oldcontext = CurrentMemoryContext; oldowner = CurrentResourceOwner; - PLy_spi_subtransaction_begin(oldcontext, oldowner); + if (!PLy_spi_subtransaction_begin(oldcontext, oldowner)) + return NULL; PG_TRY(); { @@ -403,7 +406,8 @@ PLy_cursor_fetch(PyObject *self, PyObject *args) oldcontext = CurrentMemoryContext; oldowner = CurrentResourceOwner; - PLy_spi_subtransaction_begin(oldcontext, oldowner); + if (!PLy_spi_subtransaction_begin(oldcontext, oldowner)) + return NULL; PG_TRY(); { diff --git a/src/pl/plpython/plpy_spi.c b/src/pl/plpython/plpy_spi.c index 64ca47f218..ca97f2eca6 100644 --- a/src/pl/plpython/plpy_spi.c +++ b/src/pl/plpython/plpy_spi.c @@ -77,7 +77,8 @@ PLy_spi_prepare(PyObject *self, PyObject *args) oldcontext = CurrentMemoryContext; oldowner = CurrentResourceOwner; - PLy_spi_subtransaction_begin(oldcontext, oldowner); + if (!PLy_spi_subtransaction_begin(oldcontext, oldowner)) + return NULL; PG_TRY(); { @@ -217,7 +218,8 @@ PLy_spi_execute_plan(PyObject *ob, PyObject *list, long limit) oldcontext = CurrentMemoryContext; oldowner = CurrentResourceOwner; - PLy_spi_subtransaction_begin(oldcontext, oldowner); + if (!PLy_spi_subtransaction_begin(oldcontext, oldowner)) + return NULL; PG_TRY(); { @@ -313,7 +315,8 @@ PLy_spi_execute_query(char *query, long limit) oldcontext = CurrentMemoryContext; oldowner = CurrentResourceOwner; - PLy_spi_subtransaction_begin(oldcontext, oldowner); + if (!PLy_spi_subtransaction_begin(oldcontext, oldowner)) + return NULL; PG_TRY(); { @@ -534,7 +537,9 @@ PLy_rollback(PyObject *self, PyObject *args) * MemoryContext oldcontext = CurrentMemoryContext; * ResourceOwner oldowner = CurrentResourceOwner; * - * PLy_spi_subtransaction_begin(oldcontext, oldowner); + * if (!PLy_spi_subtransaction_begin(oldcontext, oldowner)) + * return NULL; + * * PG_TRY(); * { * <call SPI functions> @@ -551,12 +556,38 @@ PLy_rollback(PyObject *self, PyObject *args) * These utilities take care of restoring connection to the SPI manager and * setting a Python exception in case of an abort. */ -void +bool PLy_spi_subtransaction_begin(MemoryContext oldcontext, ResourceOwner oldowner) { - BeginInternalSubTransaction(NULL); - /* Want to run inside function's memory context */ - MemoryContextSwitchTo(oldcontext); + PG_TRY(); + { + /* Start subtransaction (could fail) */ + BeginInternalSubTransaction(NULL); + /* Want to run inside function's memory context */ + MemoryContextSwitchTo(oldcontext); + } + PG_CATCH(); + { + ErrorData *edata; + + /* Save error info */ + MemoryContextSwitchTo(oldcontext); + edata = CopyErrorData(); + FlushErrorState(); + + /* Ensure we restore original context and owner */ + MemoryContextSwitchTo(oldcontext); + CurrentResourceOwner = oldowner; + + /* Make Python raise the exception */ + PLy_spi_exception_set(edata); + FreeErrorData(edata); + + return false; + } + PG_END_TRY(); + + return true; } void @@ -580,6 +611,8 @@ PLy_spi_subtransaction_abort(MemoryContext oldcontext, ResourceOwner oldowner) /* Abort the inner transaction */ RollbackAndReleaseCurrentSubTransaction(); + + /* Ensure we restore original context and owner */ MemoryContextSwitchTo(oldcontext); CurrentResourceOwner = oldowner; diff --git a/src/pl/plpython/plpy_spi.h b/src/pl/plpython/plpy_spi.h index 98ccd21093..a43d803928 100644 --- a/src/pl/plpython/plpy_spi.h +++ b/src/pl/plpython/plpy_spi.h @@ -22,7 +22,7 @@ typedef struct PLyExceptionEntry } PLyExceptionEntry; /* handling of SPI operations inside subtransactions */ -extern void PLy_spi_subtransaction_begin(MemoryContext oldcontext, ResourceOwner oldowner); +extern bool PLy_spi_subtransaction_begin(MemoryContext oldcontext, ResourceOwner oldowner); extern void PLy_spi_subtransaction_commit(MemoryContext oldcontext, ResourceOwner oldowner); extern void PLy_spi_subtransaction_abort(MemoryContext oldcontext, ResourceOwner oldowner); diff --git a/src/pl/plpython/sql/plpython_spi.sql b/src/pl/plpython/sql/plpython_spi.sql index fcd113acaa..69a448cd68 100644 --- a/src/pl/plpython/sql/plpython_spi.sql +++ b/src/pl/plpython/sql/plpython_spi.sql @@ -320,3 +320,22 @@ SELECT cursor_fetch_next_empty(); SELECT cursor_plan(); SELECT cursor_plan_wrong_args(); SELECT plan_composite_args(); + +-- +-- Error cases +-- + +-- Check recovery from a subtransaction start failure; to make one +-- happen repeatably, try to do a SPI operation in an allegedly +-- parallel-safe function +CREATE FUNCTION falsely_marked_parallel_safe() RETURNS SETOF int AS +$$ +plpy.execute("select 2+2") +for i in range(0, 5): + yield (i) +$$ LANGUAGE plpython3u parallel safe; + +SET debug_parallel_query = 1; +SELECT falsely_marked_parallel_safe(); +SELECT falsely_marked_parallel_safe(); +RESET debug_parallel_query; ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: [PATCH] plpython function causes server panic @ 2023-12-02 01:30 Andres Freund <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 1 reply; 4+ messages in thread From: Andres Freund @ 2023-12-02 01:30 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Hao Zhang <[email protected]>; [email protected] Hi, On 2023-12-01 20:04:15 -0500, Tom Lane wrote: > Hao Zhang <[email protected]> writes: > > I found a problem when executing the plpython function: > > After the plpython function returns an error, in the same session, if we > > continue to execute > > plpython function, the server panic will be caused. > > Thanks for the report! I see the problem is that we're not expecting > BeginInternalSubTransaction to fail. However, I'm not sure I like > this solution, mainly because it's only covering a fraction of the > problem. There are similarly unsafe usages in plperl, pltcl, and > very possibly a lot of third-party PLs. I wonder if there's a way > to deal with this issue without changing these API assumptions. There are plenty other uses, but it's not clear to me that they are similarly affected by BeginInternalSubTransaction raising an error? It e.g. doesn't immediately look like plperl's usage would be affected in a similar way? Greetings, Andres Freund ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: [PATCH] plpython function causes server panic @ 2023-12-02 01:46 Tom Lane <[email protected]> parent: Andres Freund <[email protected]> 0 siblings, 0 replies; 4+ messages in thread From: Tom Lane @ 2023-12-02 01:46 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Hao Zhang <[email protected]>; [email protected] Andres Freund <[email protected]> writes: > On 2023-12-01 20:04:15 -0500, Tom Lane wrote: >> Thanks for the report! I see the problem is that we're not expecting >> BeginInternalSubTransaction to fail. However, I'm not sure I like >> this solution, mainly because it's only covering a fraction of the >> problem. There are similarly unsafe usages in plperl, pltcl, and >> very possibly a lot of third-party PLs. I wonder if there's a way >> to deal with this issue without changing these API assumptions. > There are plenty other uses, but it's not clear to me that they are similarly > affected by BeginInternalSubTransaction raising an error? It e.g. doesn't > immediately look like plperl's usage would be affected in a similar way? Why not? We'd be longjmp'ing out from inside the Perl interpreter. Maybe Perl is so robust it doesn't care, but I'd be surprised if this can't break it. The same for Tcl. I think that plpgsql indeed doesn't care because it has no non-PG interpreter state to worry about. But it's in the minority I fear. regards, tom lane ^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2023-12-02 01:46 UTC | newest] Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]> 2023-12-02 01:04 Re: [PATCH] plpython function causes server panic Tom Lane <[email protected]> 2023-12-02 01:30 ` Re: [PATCH] plpython function causes server panic Andres Freund <[email protected]> 2023-12-02 01:46 ` Re: [PATCH] plpython function causes server panic Tom Lane <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox