Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1h27kI-00048t-U2 for pgsql-hackers@arkaria.postgresql.org; Fri, 08 Mar 2019 05:08:35 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1h27kH-0006FW-9f for pgsql-hackers@arkaria.postgresql.org; Fri, 08 Mar 2019 05:08:33 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1h27kG-0006CG-ME for pgsql-hackers@lists.postgresql.org; Fri, 08 Mar 2019 05:08:33 +0000 Received: from anastigmatix.net ([68.171.219.55]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1h27kC-0000Rl-84 for pgsql-hackers@lists.postgresql.org; Fri, 08 Mar 2019 05:08:31 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=anastigmatix.net; s=default; h=Content-Type:MIME-Version:Date:Message-ID:Cc :To:Subject:From:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=u1yP1qPtAhhHI2SOzRz9OzsaSagyK5w3/xPtfC2GklU=; b=qTrhE71AofOBSH2ST0BSIRoM9i BtgK3+eb3R7doQn/T5VXkFKIr0CH05Ay2oskcdsIHxAuNw+w8BH911qcU4lG+25F0OodUjTIyN5fR uwiEWBWQEKDX29BwRJ3L8v5oa1830+hNkZA76EkA7R9A3ZUNoTmoR0sl4VwMIUM1EyU+1n5N+fHMz uDC1m/qoaM9yqoiEIGSZDIquP34Dpxd1y/zSqd+pqibaoseqSkO7noeIxwBt/GWh68HJUFZBaVntG YIFUorhywKYRHkMS1udN5WL60d+0Cafqy0jJjtz0FqCHRf0pytO1ZrxiHAwM+t+9z3nLU3TlZIlb+ 08O8zjkg==; Received: from [184.19.63.78] (port=41167) by bay.acenet.us with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.91) (envelope-from ) id 1h27k1-00FAEo-5B; Fri, 08 Mar 2019 00:08:26 -0500 From: Chapman Flack Subject: The two "XML Fixes" patches still in need of review To: PostgreSQL Hackers Cc: Markus Winand , Alvaro Herrera , Tom Lane , Michael Paquier , Pavel Stehule X-Enigmail-Draft-Status: N1110 Message-ID: <5C81F8C0.6090901@anastigmatix.net> Date: Fri, 8 Mar 2019 00:08:16 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------080400080705050704040804" X-OutGoing-Spam-Status: No, score=-94.2 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - bay.acenet.us X-AntiAbuse: Original Domain - lists.postgresql.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - anastigmatix.net X-Get-Message-Sender-Via: bay.acenet.us: authenticated_id: chap@anastigmatix.net X-Authenticated-Sender: bay.acenet.us: chap@anastigmatix.net X-Source: X-Source-Args: X-Source-Dir: X-From-Rewrite: unmodified, already matched List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------080400080705050704040804 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Alvaro has committed two of the patches in this CF entry[1], but the remaining two have yet to attract review. This message contains only those two, just as before[2] except rebased over Alvaro's commits of the others. There are two new changes. The s added in the -docfix patch now have link text, as allowed according to b060e6c, and the -content patch now updates the definition of the 'content' type in datatypes.sgml, which I had overlooked before. xml-functions-type-docfix-3.patch adjusts the documentation of the XML type and related functions to present some behavior and limitations more clearly. xml-content-2006-2.patch changes the behavior of xmlparse and the text-to-xml cast to allow any XML 'document' (including one with a DTD) to be parsed as 'content', where the former behavior was to fail in that case. This is the same as changing the definition of XML 'content' from that of SQL:2003 to that of SQL:2006 and later. The later definition is preferable, because it eliminates a case that can fail in, e.g., pg_restore (which problem has been reported in the field). The patches apply in that order (because the -docfix one adds language describing the current 'content' behavior, then the -content one changes the behavior, and the language to match it). Regards, -Chap [1] https://commitfest.postgresql.org/22/1872/ [2] https://www.postgresql.org/message-id/3e8eab9e-7289-6c23-5e2c-153cccea2257@anastigmatix.net --------------080400080705050704040804 Content-Type: text/x-patch; name="xml-functions-type-docfix-3.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="xml-functions-type-docfix-3.patch" diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml index b462c06..94b46a6 100644 --- a/doc/src/sgml/datatype.sgml +++ b/doc/src/sgml/datatype.sgml @@ -4282,16 +4282,21 @@ SET XML OPTION { DOCUMENT | CONTENT }; SET xmloption TO { DOCUMENT | CONTENT }; The default is CONTENT, so all forms of XML - data are allowed. + data are allowed except as noted below. - With the default XML option setting, you cannot directly cast - character strings to type xml if they contain a - document type declaration, because the definition of XML content - fragment does not accept them. If you need to do that, either - use XMLPARSE or change the XML option. + In the SQL:2006 and later standard, the CONTENT form + is a proper superset of the DOCUMENT form, and so the + default XML option setting would allow casting to XML from character + strings in either form. PostgreSQL, however, + uses the SQL:2003 definition in which CONTENT form + cannot contain a document type declaration. Therefore, there is no one + setting of the XML option that will allow casting to XML from every valid + character string. The default will work almost always, but for a document + with a DTD, it will be necessary to change the XML option or + use XMLPARSE specifying DOCUMENT. diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 03859a7..0017aab 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -10140,8 +10140,13 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple + XML Functions + + XML Functions + + The functions and function-like expressions described in this section operate on values of type xml. Check . The particular behavior for individual data types is expected to evolve in order to align the - SQL and PostgreSQL data types with the XML Schema specification, - at which point a more precise description will appear. + PostgreSQL mappings with those specified in SQL:2006 and later, + as discussed in . @@ -10587,10 +10592,12 @@ SELECT xmlagg(x) FROM (SELECT * FROM test ORDER BY y DESC) AS tab; - The function xmlexists returns true if the - XPath expression in the first argument returns any nodes, and - false otherwise. (If either argument is null, the result is - null.) + The function xmlexists evaluates an XPath 1.0 + expression (the first argument), with the passed value as its context item. + The function returns false if the result of that evaluation yields an + empty nodeset, true if it yields any other value. (The function returns + null if an argument is null.) A nonnull value passed as the context item + must be an XML document, not a content fragment or any non-XML value. @@ -10607,24 +10614,12 @@ SELECT xmlexists('//town[text() = ''Toronto'']' PASSING BY VALUE 'T The BY REF or BY VALUE clauses - have no effect in PostgreSQL, but are allowed - for compatibility with other implementations. Per the SQL - standard, the one that precedes any argument is required, and indicates - the default for arguments that follow, and one may follow any argument to - override the default. - PostgreSQL ignores BY REF - and passes by value always. - - - - In the SQL standard, an xmlexists - construct evaluates an expression in the XQuery language, allows passing - values for named parameters in the expression as well as for the context - item, and does not require the passed values to be documents, or even of - XML type. - In PostgreSQL, this construct currently only - evaluates an XPath 1.0 expression, and allows passing only one value, - which must be an XML document, to be the context item. + are accepted in PostgreSQL, but ignored, as + discussed in . + In the SQL standard, an xmlexists construct first + appears in SQL:2006 and evaluates an expression in the XML Query language, + but this implementation allows only an XPath 1.0 expression, as discussed + in . @@ -10730,12 +10725,12 @@ SELECT xml_is_well_formed_document('test The function xpath_exists is a specialized form of the xpath function. Instead of returning the - individual XML values that satisfy the XPath, this function returns a - Boolean indicating whether the query was satisfied or not. This - function is equivalent to the standard XMLEXISTS predicate, + individual XML values that satisfy the XPath 1.0 expression, this function + returns a Boolean indicating whether the query was satisfied or not + (specifically, whether it produced any value other than an empty nodeset). + This function is equivalent to the XMLEXISTS predicate, except that it also offers support for a namespace mapping argument. @@ -10840,8 +10836,8 @@ SELECT xpath_exists('/my:a/text()', 'test The xmltable function produces a table based - on the given XML value, an XPath filter to extract rows, and an - optional set of column definitions. + on the given XML value, an XPath filter to extract rows, and a + set of column definitions. @@ -10852,30 +10848,33 @@ SELECT xpath_exists('/my:a/text()', 'test - The required row_expression argument is an XPath - expression that is evaluated against the supplied XML document to - obtain an ordered sequence of XML nodes. This sequence is what - xmltable transforms into output rows. + The required row_expression argument is + an XPath 1.0 expression that is evaluated, passing the + document_expression as its context item, to + obtain a set of XML nodes. These nodes are what + xmltable transforms into output rows. (No rows + will be produced if the document_expression + is null, or the row_expression produces an + empty nodeset, or any value other than a nodeset.) - document_expression provides the XML document to - operate on. - The argument must be a well-formed XML document; fragments/forests - are not accepted. + document_expression provides the context item + for the row_expression. It must be a well-formed + XML document; fragments/forests are not accepted. The BY REF or BY VALUE clauses are - accepted, as described for the xmlexists predicate, - but ignored; PostgreSQL currently passes XML by value always. + accepted but ignored, as discussed in + . + In the SQL standard, an xmltable construct first + appears in SQL:2006 and evaluates expressions in the XML Query language, + but this implementation allows only XPath 1.0 expressions, as discussed + in . The mandatory COLUMNS clause specifies the list of columns in the output table. - If the COLUMNS clause is omitted, the rows in the result - set contain a single column of type xml containing the - data matched by row_expression. - If COLUMNS is specified, each entry describes a - single column. + Each entry describes a single column. See the syntax summary above for the format. The column name and type are required; the path, default and nullability clauses are optional. @@ -10889,42 +10888,78 @@ SELECT xpath_exists('/my:a/text()', 'test - The column_expression for a column is an XPath expression - that is evaluated for each row, relative to the result of the - row_expression, to find the value of the column. + The column_expression for a column is an XPath 1.0 + expression + that is evaluated for each row, with the current node from the + row_expression result as its context item, + to find the value of the column. If no column_expression is given, then the column name is used as an implicit path. - If a column's XPath expression returns multiple elements, an error - is raised. - If the expression matches an empty tag, the result is an - empty string (not NULL). - Any xsi:nil attributes are ignored. + If a column's XPath expression returns a non-XML value (limited to + string, boolean, or double in XPath 1.0) and the column has a + PostgreSQL type other than xml, the column will be set + as if by assigning the value's string representation to the PostgreSQL + type (adjusting the "string representation" of a boolean to + 1 or 0 if the target column type + category is numeric, otherwise true or + false). + + + + If the column's expression returns a non-empty set of XML nodes + and the target column's type is xml, the column will + be assigned the expression result exactly, if it is of document or + content form. + + + A result containing more than one element node at the top level, or + non-whitespace text outside of an element, is an example of content form. + An XPath result can be of neither form, for example if it returns an + attribute node selected from the element that contains it. Such a result + will be put into content form with each such disallowed node replaced by + its string value, as defined for the XPath 1.0 + string function. + + + + + + A non-XML result assigned to an xml output column produces + content, a single text node with the string value of the result. + An XML result assigned to a column of any other type may not have more than + one node, or an error is raised. If there is exactly one node, the column + will be set as if by assigning the node's string + value (as defined for the XPath 1.0 string function) + to the PostgreSQL type. - The text body of the XML matched by the column_expression - is used as the column value. Multiple text() nodes - within an element are concatenated in order. Any child elements, - processing instructions, and comments are ignored, but the text contents - of child elements are concatenated to the result. + The string value of an XML element is the concatenation, in document order, + of all text nodes contained in that element and its descendants. The string + value of an element with no descendant text nodes is an + empty string (not NULL). + Any xsi:nil attributes are ignored. Note that the whitespace-only text() node between two non-text elements is preserved, and that leading whitespace on a text() node is not flattened. + The XPath 1.0 string function may be consulted for the + rules defining the string value of other XML node types and non-XML values. + + + + The conversion rules presented here are not exactly those of the SQL + standard, as discussed in . - If the path expression does not match for a given row but - default_expression is specified, the value resulting - from evaluating that expression is used. - If no DEFAULT clause is given for the column, - the field will be set to NULL. - It is possible for a default_expression to reference - the value of output columns that appear prior to it in the column list, - so the default of one column may be based on the value of another - column. + If the path expression returns an empty nodeset + (typically, when it does not match) + for a given row, the column will be set to NULL, unless + a default_expression is specified; then the + value resulting from evaluating that expression is used. @@ -10936,20 +10971,14 @@ SELECT xpath_exists('/my:a/text()', 'test - Unlike regular PostgreSQL functions, column_expression - and default_expression are not evaluated to a simple - value before calling the function. - column_expression is normally evaluated - exactly once per input row, and default_expression - is evaluated each time a default is needed for a field. - If the expression qualifies as stable or immutable the repeat + A default_expression, rather than being + evaluated immediately when xmltable is called, + is evaluated each time a default is needed for the column. + If the expression qualifies as stable or immutable, the repeat evaluation may be skipped. - Effectively xmltable behaves more like a subquery than a - function call. This means that you can usefully use volatile functions like - nextval in default_expression, and - column_expression may depend on other parts of the - XML document. + nextval in + default_expression. @@ -11297,6 +11326,293 @@ table2-mapping ]]> + + + Limits and Compatibility + + + XML Functions + limits and compatibility + + + + Queries restricted to XPath 1.0 + + + The PostgreSQL-specific functions + xpath and xpath_exists query + XML documents using the XPath language, and + PostgreSQL also provides XPath-only variants of + the standard functions XMLEXISTS and + XMLTABLE, which are defined in the SQL standard to use + the XQuery language. For all of these functions, + PostgreSQL relies on the + libxml library, which provides only XPath 1.0. + + + + There is a strong connection between the XQuery language and XPath versions + 2.0 and later: any expression that is syntactically valid and executes + successfully in both produces the same result (with a minor exception for + expressions containing numeric character references or predefined entity + references, which XQuery replaces with the corresponding character while + XPath leaves them alone). But there is no such connection between XPath 1.0 + and XQuery or the later XPath versions; it was an earlier language and + differs in many respects. + + + + There are two categories of limitation to keep in mind: the restriction + from XQuery to XPath for the functions specified in the SQL standard, and + the restriction of XPath to version 1.0 for both the standard and the + PostgreSQL-specific functions. + + + + Restriction of XQuery to XPath + + + Features of XQuery beyond those of XPath include: + + + + + XQuery expressions can construct and return new XML nodes, in addition + to all possible XPath values. XPath can introduce and return values of + the atomic types (numbers, strings, and so on) but can only return XML + nodes already present in documents supplied as input to the expression. + + + + + + XQuery has control constructs for iteration, sorting, and grouping. + + + + + + XQuery allows the declaration and use of local functions. + + + + + + + Recent XPath versions begin to offer overlapping capabilities + (such as the functional-style for-each and + sort, anonymous functions, and + parse-xml to create a node from a string), + but these were not available before XPath 3.0. + + + + + Restriction of XPath to 1.0 + + + For developers familiar with XQuery and XPath 2.0 or later, or porting + queries from other systems, XPath 1.0 presents a number of differences to + contend with: + + + + + The fundamental type of an XQuery/XPath expression, the + sequence, which can contain XML nodes, atomic values, + or both, does not exist in XPath 1.0. A 1.0 expression can only produce + a nodeset (possibly empty, or with one XML node or more), or a single + atomic value. + + + + + + Unlike an XQuery/XPath sequence, which can contain any desired items + in any desired order, an XPath 1.0 nodeset has no guaranteed order and, + like any set, can have no member appear more than once. (The + libxml library does seem to always return + nodesets to PostgreSQL with their members + in the same relative order they had in the input document; it does not + commit to this behavior, and an XPath 1.0 expression cannot control + it.) + + + + + + While XQuery/XPath provides all of the types defined in XML Schema + and many operators and functions over those types, XPath 1.0 has only + nodesets and three atomic types, boolean, + double, and string. + + + + + + XPath 1.0 has no conditional operator. An XQuery/XPath expression + such as if ( hat ) then hat/@size else "no hat" + has no XPath 1.0 equivalent. + + + + + + XPath 1.0 has no ordering comparison operator for strings. Both + "cat" < "dog" and + "cat" > "dog" are false, because each is a + numeric comparison of two NaNs. In contrast, + = and != do compare the strings + as strings. + + + + + + XPath 1.0 blurs the distinction between + value comparisons and + general comparisons as XQuery/XPath define them. + Both sale/@hatsize = 7 and + sale/@customer = "alice" are existentially + quantified comparisons, true if there is any sale with the given value + for the attribute, but sale/@taxable = false() + is a value comparison to the + effective boolean value of a whole nodeset, + and true only if no sale has a taxable attribute + at all. + + + + + + In the XQuery/XPath data model, a document node + can have either document form (exactly one top-level element, only + comments and processing instructions outside of it) or content form + (with those constraints relaxed). Its equivalent in XPath 1.0, the + root node, can only be in document form. + This is part of the reason an xml value passed as the + context item to any PostgreSQL XPath-based + function must be in document form. + + + + + + + The differences highlighted here are not all of them. In XQuery and + the 2.0 and later versions of XPath, there is an XPath 1.0 compatibility + mode, and the W3C lists of function library + changes + and language + changes + applied in that mode offer a more complete account of the + differences, if still not exhaustive; the compatibility mode cannot make + the later languages exactly equivalent to XPath 1.0. + + + + + Mappings between SQL and XML data types and values + + + In SQL:2006 and later, both directions of conversion between standard SQL + data types and the XML Schema types are specified precisely. However, the + rules are expressed using the types and semantics of XQuery/XPath, and + have no direct application to the different data model of XPath 1.0. + + + + When PostgreSQL maps SQL data values to XML + (as in xmlelement), or XML to SQL (as in the output + columns of xmltable), except for the few cases + treated specially, PostgreSQL simply assumes + that the XML data type's XPath 1.0 string form will be valid as the + text-input form of the SQL datatype, and conversely. This rule has the + virtue of simplicity while producing, for many data types, results similar + to the mappings specified in the standard. + + + + Where interoperability with other systems is a concern, for some data + types, it may be necessary to use available data type formatting functions + (such as those in ) explicitly in + queries to produce the standard mappings. + + + + + + + Incidental limits of the implementation + + + + This section concerns limits that are not inherent in the + libxml library, but apply to the current + implementation in PostgreSQL. + + + + Only <literal>BY VALUE</literal> passing mechanism supported + + + The SQL standard defines two passing mechanisms + that apply when passing an XML argument from SQL to an XML function or + receiving a result: BY REF, in which a particular XML + value retains its node identity, and BY VALUE, in which + the content of the XML is passed but node identity is not preserved. A + mechanism can be specified before a list of parameters, as the default + mechanism for all of them, or after any parameter, to override the + default. + + + + To illustrate the difference, if + x is an XML value, these two queries in + an SQL:2006 environment would produce true and false, respectively: + + + + + + PostgreSQL will accept either + BY VALUE or BY REF in an + XMLEXISTS or XMLTABLE construct, + but ignores them; the xml data type holds a character-string + serialized representation, so there is no node identity to preserve, + and passing is always BY VALUE. + + + + + Cannot pass named parameters to queries + + + The XPath-based functions support passing one parameter to serve as the + XPath expression's context item, but do not support passing additional + values to be available to the expression as named parameters. + + + + + No <type>XML(SEQUENCE)</type> type + + + The PostgreSQL xml can only hold + a value in DOCUMENT or CONTENT form. + An XQuery/XPath expression context item must be a single XML node + or atomic value, while XPath 1.0 further restricts it to only an XML node, + and has no node type allowing CONTENT. The upshot is + that a well-formed DOCUMENT is the only form of XML + value that PostgreSQL can supply as an XPath + context item. + + + + diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt index bade0fe..e947613 100644 --- a/src/backend/catalog/sql_features.txt +++ b/src/backend/catalog/sql_features.txt @@ -569,7 +569,7 @@ X056 Advanced table mapping: data mapping YES X057 Advanced table mapping: metadata mapping YES X058 Advanced table mapping: base64 encoding of binary strings YES X059 Advanced table mapping: hex encoding of binary strings YES -X060 XMLParse: character string input and CONTENT option YES +X060 XMLParse: character string input and CONTENT option YES uses SQL:2003 definition, so a document with a DTD cannot be parsed as CONTENT X061 XMLParse: character string input and DOCUMENT option YES X065 XMLParse: BLOB input and CONTENT option NO X066 XMLParse: BLOB input and DOCUMENT option NO @@ -593,7 +593,7 @@ X085 Predefined namespace prefixes NO X086 XML namespace declarations in XMLTable NO X090 XML document predicate YES X091 XML content predicate NO -X096 XMLExists NO XPath only +X096 XMLExists NO XPath 1.0 only X100 Host language support for XML: CONTENT option NO X101 Host language support for XML: DOCUMENT option NO X110 Host language support for XML: VARCHAR mapping NO @@ -661,11 +661,11 @@ X282 XMLValidate with CONTENT option NO X283 XMLValidate with SEQUENCE option NO X284 XMLValidate: NAMESPACE without ELEMENT clause NO X286 XMLValidate: NO NAMESPACE with ELEMENT clause NO -X300 XMLTable NO XPath only +X300 XMLTable NO XPath 1.0 only X301 XMLTable: derived column list option YES X302 XMLTable: ordinality column option YES X303 XMLTable: column default option YES -X304 XMLTable: passing a context item YES +X304 XMLTable: passing a context item YES must be XML DOCUMENT X305 XMLTable: initializing an XQuery variable NO X400 Name and identifier mapping YES X410 Alter column data type: XML type YES --------------080400080705050704040804 Content-Type: text/x-patch; name="xml-content-2006-2.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="xml-content-2006-2.patch" diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml index 94b46a6..752eadf 100644 --- a/doc/src/sgml/datatype.sgml +++ b/doc/src/sgml/datatype.sgml @@ -4208,9 +4208,11 @@ a0ee-bc99-9c0b-4ef8-bb6d-6bb9-bd38-0a11 The xml type can store well-formed documents, as defined by the XML standard, as well - as content fragments, which are defined by the - production XMLDecl? content in the XML - standard. Roughly, this means that content fragments can have + as content fragments, which are defined by reference to the + more permissive + document node + of the XQuery and XPath data model. + Roughly, this means that content fragments can have more than one top-level element or character node. The expression xmlvalue IS DOCUMENT can be used to evaluate whether a particular xml @@ -4282,24 +4284,9 @@ SET XML OPTION { DOCUMENT | CONTENT }; SET xmloption TO { DOCUMENT | CONTENT }; The default is CONTENT, so all forms of XML - data are allowed except as noted below. + data are allowed. - - - In the SQL:2006 and later standard, the CONTENT form - is a proper superset of the DOCUMENT form, and so the - default XML option setting would allow casting to XML from character - strings in either form. PostgreSQL, however, - uses the SQL:2003 definition in which CONTENT form - cannot contain a document type declaration. Therefore, there is no one - setting of the XML option that will allow casting to XML from every valid - character string. The default will work almost always, but for a document - with a DTD, it will be necessary to change the XML option or - use XMLPARSE specifying DOCUMENT. - - - diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt index e947613..32908c1 100644 --- a/src/backend/catalog/sql_features.txt +++ b/src/backend/catalog/sql_features.txt @@ -569,7 +569,7 @@ X056 Advanced table mapping: data mapping YES X057 Advanced table mapping: metadata mapping YES X058 Advanced table mapping: base64 encoding of binary strings YES X059 Advanced table mapping: hex encoding of binary strings YES -X060 XMLParse: character string input and CONTENT option YES uses SQL:2003 definition, so a document with a DTD cannot be parsed as CONTENT +X060 XMLParse: character string input and CONTENT option YES X061 XMLParse: character string input and DOCUMENT option YES X065 XMLParse: BLOB input and CONTENT option NO X066 XMLParse: BLOB input and DOCUMENT option NO diff --git a/src/backend/utils/adt/xml.c b/src/backend/utils/adt/xml.c index 28b3eaa..23f7dcb 100644 --- a/src/backend/utils/adt/xml.c +++ b/src/backend/utils/adt/xml.c @@ -110,6 +110,8 @@ struct PgXmlErrorContext /* current error status and accumulated message, if any */ bool err_occurred; StringInfoData err_buf; + /* set in xml_parse if trying to parse CONTENT; reset in err hdlr if DTD */ + bool tentatively_content; /* previous libxml error handling state (saved by pg_xml_init) */ xmlStructuredErrorFunc saved_errfunc; void *saved_errcxt; @@ -120,6 +122,7 @@ struct PgXmlErrorContext static xmlParserInputPtr xmlPgEntityLoader(const char *URL, const char *ID, xmlParserCtxtPtr ctxt); static void xml_errorHandler(void *data, xmlErrorPtr error); +static bool xml_onlyPrologNodes(xmlDocPtr doc); static void xml_ereport_by_code(int level, int sqlcode, const char *msg, int errcode); static void chopStringInfoNewlines(StringInfo str); @@ -1002,6 +1005,7 @@ pg_xml_init(PgXmlStrictness strictness) errcxt->magic = ERRCXT_MAGIC; errcxt->strictness = strictness; errcxt->err_occurred = false; + errcxt->tentatively_content = false; initStringInfo(&errcxt->err_buf); /* @@ -1433,6 +1437,9 @@ xml_parse(text *data, XmlOptionType xmloption_arg, bool preserve_whitespace, xmlChar *string; xmlChar *utf8string; PgXmlErrorContext *xmlerrcxt; + int errcode_document_parse = ERRCODE_INVALID_XML_DOCUMENT; + char const *errmsg_document_parse = "invalid XML document"; + char const *errmsg_content_parse = "invalid XML content"; volatile xmlParserCtxtPtr ctxt = NULL; volatile xmlDocPtr doc = NULL; @@ -1457,6 +1464,7 @@ xml_parse(text *data, XmlOptionType xmloption_arg, bool preserve_whitespace, xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, "could not allocate parser context"); +try_other_xmloption: if (xmloption_arg == XMLOPTION_DOCUMENT) { /* @@ -1472,8 +1480,8 @@ xml_parse(text *data, XmlOptionType xmloption_arg, bool preserve_whitespace, XML_PARSE_NOENT | XML_PARSE_DTDATTR | (preserve_whitespace ? 0 : XML_PARSE_NOBLANKS)); if (doc == NULL || xmlerrcxt->err_occurred) - xml_ereport(xmlerrcxt, ERROR, ERRCODE_INVALID_XML_DOCUMENT, - "invalid XML document"); + xml_ereport(xmlerrcxt, ERROR, errcode_document_parse, + errmsg_document_parse); } else { @@ -1497,11 +1505,42 @@ xml_parse(text *data, XmlOptionType xmloption_arg, bool preserve_whitespace, /* allow empty content */ if (*(utf8string + count)) { + xmlerrcxt->tentatively_content = true; res_code = xmlParseBalancedChunkMemory(doc, NULL, NULL, 0, utf8string + count, NULL); + /* + * If tentatively_content is now false, the error handler has + * reset it, which will only happen for one reason: a DTD was + * found in the input. The SQL/XML:2003 definition of CONTENT + * is not satisfied by a document with a DTD, which is a bit of + * a wart, as it means the CONTENT type is not a proper superset + * of DOCUMENT. SQL/XML:2006 and later fix that, by declaring + * that any DOCUMENT value is indeed also a CONTENT value. That + * definition is more useful, as CONTENT becomes usable for + * parsing input of unknown form (think pg_restore). Without + * bringing the whole implementation along to 2006+, we can + * provide the 2006+ definition of CONTENT easily enough, by + * catching this error case and simply retrying the parse + * as DOCUMENT. + */ + if ( ! xmlerrcxt->tentatively_content ) + { + /* + * xmlParseBalanced... was not passed our ctxt; it + * creates one internally for its own use, so we have + * no need to clear/reset ctxt here. + */ + xmlFreeDoc(doc); + xmloption_arg = XMLOPTION_DOCUMENT; + errcode_document_parse = ERRCODE_INVALID_XML_CONTENT; + errmsg_document_parse = errmsg_content_parse; + goto try_other_xmloption; + } if (res_code != 0 || xmlerrcxt->err_occurred) + { xml_ereport(xmlerrcxt, ERROR, ERRCODE_INVALID_XML_CONTENT, - "invalid XML content"); + errmsg_content_parse); + } } } } @@ -1699,6 +1738,27 @@ xml_errorHandler(void *data, xmlErrorPtr error) switch (domain) { case XML_FROM_PARSER: + /* + * One very specific error from the parser will be intercepted here + * if strictness is PG_XML_STRICTNESS_WELLFORMED (meaning the call + * originates in xml_parse), the input is being tentatively parsed + * as content, and the parser has stumbled on what looks like a DTD. + * For that case, just change tentatively_content to false and + * return, and xml_parse will retry the input as a document. + * See xml_parse for motivation. + */ + if (xmlerrcxt->strictness == PG_XML_STRICTNESS_WELLFORMED + && xmlerrcxt->tentatively_content + && error->code == XML_ERR_NAME_REQUIRED + && input != NULL + && input->cur != NULL + && 0 == xmlStrncmp(input->cur, BAD_CAST"!DOCTYPE", 8) + && xml_onlyPrologNodes(ctxt->myDoc)) + { + xmlerrcxt->tentatively_content = false; + return; + } + /* FALLTHROUGH */ case XML_FROM_NONE: case XML_FROM_MEMORY: case XML_FROM_IO: @@ -1811,6 +1871,52 @@ xml_errorHandler(void *data, xmlErrorPtr error) pfree(errorBuf); } +/* + * Check (from the error handler) that the document node's only children already + * parsed are those that can precede a DTD. + * + * To deliver the SQL/XML:2006+ definition of 'content', the error handler above + * can cause a reparse as 'document' if a tentative parse as 'content' trips + * over something that looks like a DTD. Without this check, goofy input like + * could also cause an attempted 'document' reparse, which + * would correctly fail, but the error message could give the wrong reason. + * + * This explores the document that is being constructed. If anything about its + * structure is not as expected, the return is false, so the libxml error will + * be reported, with no other attempt at special handling. + */ +static bool +xml_onlyPrologNodes(xmlDocPtr doc) +{ + xmlNodePtr n; + + if ( doc == NULL ) + return false; + + n = doc->children; /* parseBalancedChunk creates a temporary "pseudoroot" */ + if ( n == NULL || n->type != XML_ELEMENT_NODE ) + return false; + if ( n->name == NULL || 0 != xmlStrcmp(n->name, BAD_CAST"pseudoroot") ) + return false; + + for ( n = n->children; n != NULL; n = n->next ) + { + switch (n->type) + { + case XML_PI_NODE: + case XML_COMMENT_NODE: + continue; + case XML_TEXT_NODE: + if ( xmlIsBlankNode(n) ) + continue; + /* FALLTHROUGH */ + default: + return false; + } + } + return true; +} + /* * Wrapper for "ereport" function for XML-related errors. The "msg" diff --git a/src/test/regress/expected/xml.out b/src/test/regress/expected/xml.out index 2085fa0..0aae600 100644 --- a/src/test/regress/expected/xml.out +++ b/src/test/regress/expected/xml.out @@ -532,6 +532,13 @@ LINE 1: EXECUTE foo ('bad'); DETAIL: line 1: Start tag expected, '<' not found bad ^ +SELECT xml ''; +ERROR: invalid XML document +LINE 1: SELECT xml ''; + ^ +DETAIL: line 1: Extra content at the end of the document + + ^ SET XML OPTION CONTENT; EXECUTE foo (''); xmlconcat @@ -545,6 +552,45 @@ EXECUTE foo ('good'); good (1 row) +SELECT xml ' '; + xml +-------------------------------------------------------------------- + +(1 row) + +SELECT xml ' '; + xml +------------------------------ + +(1 row) + +SELECT xml ''; + xml +------------------ + +(1 row) + +SELECT xml ' oops '; +ERROR: invalid XML content +LINE 1: SELECT xml ' oops '; + ^ +DETAIL: line 1: StartTag: invalid element name + oops + ^ +SELECT xml ' '; +ERROR: invalid XML content +LINE 1: SELECT xml ' '; + ^ +DETAIL: line 1: StartTag: invalid element name + + ^ +SELECT xml ''; +ERROR: invalid XML content +LINE 1: SELECT xml ''; + ^ +DETAIL: line 1: Extra content at the end of the document + + ^ -- Test backwards parsing CREATE VIEW xmlview1 AS SELECT xmlcomment('test'); CREATE VIEW xmlview2 AS SELECT xmlconcat('hello', 'you'); diff --git a/src/test/regress/expected/xml_1.out b/src/test/regress/expected/xml_1.out index 7f86696..d1a03b5 100644 --- a/src/test/regress/expected/xml_1.out +++ b/src/test/regress/expected/xml_1.out @@ -429,11 +429,53 @@ EXECUTE foo (''); ERROR: prepared statement "foo" does not exist EXECUTE foo ('bad'); ERROR: prepared statement "foo" does not exist +SELECT xml ''; +ERROR: unsupported XML feature +LINE 1: SELECT xml ''; + ^ +DETAIL: This functionality requires the server to be built with libxml support. +HINT: You need to rebuild PostgreSQL using --with-libxml. SET XML OPTION CONTENT; EXECUTE foo (''); ERROR: prepared statement "foo" does not exist EXECUTE foo ('good'); ERROR: prepared statement "foo" does not exist +SELECT xml ' '; +ERROR: unsupported XML feature +LINE 1: SELECT xml ' '; +ERROR: unsupported XML feature +LINE 1: SELECT xml ' '; +ERROR: unsupported XML feature +LINE 1: SELECT xml ''; + ^ +DETAIL: This functionality requires the server to be built with libxml support. +HINT: You need to rebuild PostgreSQL using --with-libxml. +SELECT xml ' oops '; +ERROR: unsupported XML feature +LINE 1: SELECT xml ' oops '; + ^ +DETAIL: This functionality requires the server to be built with libxml support. +HINT: You need to rebuild PostgreSQL using --with-libxml. +SELECT xml ' '; +ERROR: unsupported XML feature +LINE 1: SELECT xml ' '; + ^ +DETAIL: This functionality requires the server to be built with libxml support. +HINT: You need to rebuild PostgreSQL using --with-libxml. +SELECT xml ''; +ERROR: unsupported XML feature +LINE 1: SELECT xml ''; + ^ +DETAIL: This functionality requires the server to be built with libxml support. +HINT: You need to rebuild PostgreSQL using --with-libxml. -- Test backwards parsing CREATE VIEW xmlview1 AS SELECT xmlcomment('test'); CREATE VIEW xmlview2 AS SELECT xmlconcat('hello', 'you'); diff --git a/src/test/regress/expected/xml_2.out b/src/test/regress/expected/xml_2.out index 510b09b..9756b6e 100644 --- a/src/test/regress/expected/xml_2.out +++ b/src/test/regress/expected/xml_2.out @@ -512,6 +512,13 @@ LINE 1: EXECUTE foo ('bad'); DETAIL: line 1: Start tag expected, '<' not found bad ^ +SELECT xml ''; +ERROR: invalid XML document +LINE 1: SELECT xml ''; + ^ +DETAIL: line 1: Extra content at the end of the document + + ^ SET XML OPTION CONTENT; EXECUTE foo (''); xmlconcat @@ -525,6 +532,45 @@ EXECUTE foo ('good'); good (1 row) +SELECT xml ' '; + xml +-------------------------------------------------------------------- + +(1 row) + +SELECT xml ' '; + xml +------------------------------ + +(1 row) + +SELECT xml ''; + xml +------------------ + +(1 row) + +SELECT xml ' oops '; +ERROR: invalid XML content +LINE 1: SELECT xml ' oops '; + ^ +DETAIL: line 1: StartTag: invalid element name + oops + ^ +SELECT xml ' '; +ERROR: invalid XML content +LINE 1: SELECT xml ' '; + ^ +DETAIL: line 1: StartTag: invalid element name + + ^ +SELECT xml ''; +ERROR: invalid XML content +LINE 1: SELECT xml ''; + ^ +DETAIL: line 1: Extra content at the end of the document + + ^ -- Test backwards parsing CREATE VIEW xmlview1 AS SELECT xmlcomment('test'); CREATE VIEW xmlview2 AS SELECT xmlconcat('hello', 'you'); diff --git a/src/test/regress/sql/xml.sql b/src/test/regress/sql/xml.sql index 8057a46..71431d8 100644 --- a/src/test/regress/sql/xml.sql +++ b/src/test/regress/sql/xml.sql @@ -149,10 +149,17 @@ PREPARE foo (xml) AS SELECT xmlconcat('', $1); SET XML OPTION DOCUMENT; EXECUTE foo (''); EXECUTE foo ('bad'); +SELECT xml ''; SET XML OPTION CONTENT; EXECUTE foo (''); EXECUTE foo ('good'); +SELECT xml ' '; +SELECT xml ' '; +SELECT xml ''; +SELECT xml ' oops '; +SELECT xml ' '; +SELECT xml ''; -- Test backwards parsing --------------080400080705050704040804--