Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1h5eJM-0007iQ-Qh for pgsql-hackers@arkaria.postgresql.org; Sun, 17 Mar 2019 22:31:21 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1h5eJJ-0001BW-Fe for pgsql-hackers@arkaria.postgresql.org; Sun, 17 Mar 2019 22:31:17 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1h5eJI-0001BO-GY for pgsql-hackers@lists.postgresql.org; Sun, 17 Mar 2019 22:31:17 +0000 Received: from anastigmatix.net ([68.171.219.55]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1h5eJD-0002Nv-Ii for pgsql-hackers@postgresql.org; Sun, 17 Mar 2019 22:31:15 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=anastigmatix.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:Cc:References:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=RZ/uvb1hRE1bT6kGc9r9Hh+Xiaq3Swhj41KBnqTKGU4=; b=fUbVtgoUbydUb95ZvH0UU/+da wKIsNFGMeL7h+u9FnjQVRhaO40XJOFjE/LDELoI+l9fhafFK9INNdWpsuv1JK9MOfMXi/CsuNo2zR cJk1wYBamGz/1L1qCMM2YfOS4AS7goZSXFBUVksghP1Hi4c+Rjvnk2YlSYA/CzEoKgp81IXfmSRAB kCUcIXkICcnzL0Agzg3+WPD7r+DMlJg1bmWZwe0pJAP8wPVgGsU9eTHzqIC2UQQLVo0jIBwXP4tg2 doTFh0XwImit1Hcozl/YmdaJ5jprlo+AuZBVZ/wix1MEwoM/d8gZpQOiAAx3ENGlI03O97rhP3gj5 6mXTszjDQ==; Received: from [184.19.63.78] (port=46378) by bay.acenet.us with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.91) (envelope-from ) id 1h5eJ3-001BJd-1n; Sun, 17 Mar 2019 18:31:09 -0400 Subject: Re: Fix XML handling with DOCTYPE To: Tom Lane References: <5C8D5D16.7090206@anastigmatix.net> <23319.1552769738@sss.pgh.pa.us> <5C8D6681.4070504@anastigmatix.net> <24203.1552771272@sss.pgh.pa.us> <5C8D79AF.7010204@anastigmatix.net> <5146.1552837534@sss.pgh.pa.us> <5C8E7FC0.8040401@anastigmatix.net> <31664.1552842991@sss.pgh.pa.us> <5C8E8E2F.9050600@anastigmatix.net> <25228.1552849567@sss.pgh.pa.us> Cc: Ryan Lambert , pgsql-hackers@postgresql.org From: Chapman Flack X-Enigmail-Draft-Status: N1110 Message-ID: <5C8ECAA4.3090301@anastigmatix.net> Date: Sun, 17 Mar 2019 18:31:00 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: <25228.1552849567@sss.pgh.pa.us> Content-Type: multipart/mixed; boundary="------------080405040605080502000109" X-OutGoing-Spam-Status: No, score=-94.2 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - bay.acenet.us X-AntiAbuse: Original Domain - postgresql.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - anastigmatix.net X-Get-Message-Sender-Via: bay.acenet.us: authenticated_id: chap@anastigmatix.net X-Authenticated-Sender: bay.acenet.us: chap@anastigmatix.net X-Source: X-Source-Args: X-Source-Dir: X-From-Rewrite: unmodified, already matched List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------080405040605080502000109 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit On 03/17/19 15:06, Tom Lane wrote: > The error message issue is indeed a concern, but I don't see why it's > complicated if you agree that > >> If the query asked for CONTENT, any error result should be one you could get >> when parsing as CONTENT. > > That just requires us to save the first error message and be sure to issue > that one not the DOCUMENT one, no? I confess I haven't looked hard yet at how to do that. The way errors come out of libxml is more involved than "here's a message", so there's a choice of (a) try to copy off that struct in a way that's sure to survive re-executing the parser, and then use the copy, or (b) generate a message right away from the structured information and save that, and I guess b might not be too bad; a might not be too bad, or it might, and slide right back into the kind of libxml-behavior-assumptions you're wanting to avoid. Meanwhile, here is a patch on the lines I proposed earlier, with a pre-check. Any performance hit that it could entail (which I'd really expect to be de minimis, though I haven't benchmarked) ought to be compensated by the strlen I changed to strnlen in parse_xml_decl (as there's really no need to run off and count the whole rest of the input just to know if 1, 2, 3, or 4 bytes are available to decode a UTF-8 char). ... and, yes, I know that could be an independent patch, and then the performance effect here should be measured from there. But it was near what I was doing anyway, so I included it here. Attaching both still-outstanding patches (this one and docfix) so the CF app doesn't lose one. Regards, -Chap --------------080405040605080502000109 Content-Type: text/x-patch; name="xml-functions-type-docfix-3.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="xml-functions-type-docfix-3.patch" diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml index b462c06..94b46a6 100644 --- a/doc/src/sgml/datatype.sgml +++ b/doc/src/sgml/datatype.sgml @@ -4282,16 +4282,21 @@ SET XML OPTION { DOCUMENT | CONTENT }; SET xmloption TO { DOCUMENT | CONTENT }; The default is CONTENT, so all forms of XML - data are allowed. + data are allowed except as noted below. - With the default XML option setting, you cannot directly cast - character strings to type xml if they contain a - document type declaration, because the definition of XML content - fragment does not accept them. If you need to do that, either - use XMLPARSE or change the XML option. + In the SQL:2006 and later standard, the CONTENT form + is a proper superset of the DOCUMENT form, and so the + default XML option setting would allow casting to XML from character + strings in either form. PostgreSQL, however, + uses the SQL:2003 definition in which CONTENT form + cannot contain a document type declaration. Therefore, there is no one + setting of the XML option that will allow casting to XML from every valid + character string. The default will work almost always, but for a document + with a DTD, it will be necessary to change the XML option or + use XMLPARSE specifying DOCUMENT. diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 03859a7..0017aab 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -10140,8 +10140,13 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple + XML Functions + + XML Functions + + The functions and function-like expressions described in this section operate on values of type xml. Check . The particular behavior for individual data types is expected to evolve in order to align the - SQL and PostgreSQL data types with the XML Schema specification, - at which point a more precise description will appear. + PostgreSQL mappings with those specified in SQL:2006 and later, + as discussed in . @@ -10587,10 +10592,12 @@ SELECT xmlagg(x) FROM (SELECT * FROM test ORDER BY y DESC) AS tab; - The function xmlexists returns true if the - XPath expression in the first argument returns any nodes, and - false otherwise. (If either argument is null, the result is - null.) + The function xmlexists evaluates an XPath 1.0 + expression (the first argument), with the passed value as its context item. + The function returns false if the result of that evaluation yields an + empty nodeset, true if it yields any other value. (The function returns + null if an argument is null.) A nonnull value passed as the context item + must be an XML document, not a content fragment or any non-XML value. @@ -10607,24 +10614,12 @@ SELECT xmlexists('//town[text() = ''Toronto'']' PASSING BY VALUE 'T The BY REF or BY VALUE clauses - have no effect in PostgreSQL, but are allowed - for compatibility with other implementations. Per the SQL - standard, the one that precedes any argument is required, and indicates - the default for arguments that follow, and one may follow any argument to - override the default. - PostgreSQL ignores BY REF - and passes by value always. - - - - In the SQL standard, an xmlexists - construct evaluates an expression in the XQuery language, allows passing - values for named parameters in the expression as well as for the context - item, and does not require the passed values to be documents, or even of - XML type. - In PostgreSQL, this construct currently only - evaluates an XPath 1.0 expression, and allows passing only one value, - which must be an XML document, to be the context item. + are accepted in PostgreSQL, but ignored, as + discussed in . + In the SQL standard, an xmlexists construct first + appears in SQL:2006 and evaluates an expression in the XML Query language, + but this implementation allows only an XPath 1.0 expression, as discussed + in . @@ -10730,12 +10725,12 @@ SELECT xml_is_well_formed_document('test The function xpath_exists is a specialized form of the xpath function. Instead of returning the - individual XML values that satisfy the XPath, this function returns a - Boolean indicating whether the query was satisfied or not. This - function is equivalent to the standard XMLEXISTS predicate, + individual XML values that satisfy the XPath 1.0 expression, this function + returns a Boolean indicating whether the query was satisfied or not + (specifically, whether it produced any value other than an empty nodeset). + This function is equivalent to the XMLEXISTS predicate, except that it also offers support for a namespace mapping argument. @@ -10840,8 +10836,8 @@ SELECT xpath_exists('/my:a/text()', 'test The xmltable function produces a table based - on the given XML value, an XPath filter to extract rows, and an - optional set of column definitions. + on the given XML value, an XPath filter to extract rows, and a + set of column definitions. @@ -10852,30 +10848,33 @@ SELECT xpath_exists('/my:a/text()', 'test - The required row_expression argument is an XPath - expression that is evaluated against the supplied XML document to - obtain an ordered sequence of XML nodes. This sequence is what - xmltable transforms into output rows. + The required row_expression argument is + an XPath 1.0 expression that is evaluated, passing the + document_expression as its context item, to + obtain a set of XML nodes. These nodes are what + xmltable transforms into output rows. (No rows + will be produced if the document_expression + is null, or the row_expression produces an + empty nodeset, or any value other than a nodeset.) - document_expression provides the XML document to - operate on. - The argument must be a well-formed XML document; fragments/forests - are not accepted. + document_expression provides the context item + for the row_expression. It must be a well-formed + XML document; fragments/forests are not accepted. The BY REF or BY VALUE clauses are - accepted, as described for the xmlexists predicate, - but ignored; PostgreSQL currently passes XML by value always. + accepted but ignored, as discussed in + . + In the SQL standard, an xmltable construct first + appears in SQL:2006 and evaluates expressions in the XML Query language, + but this implementation allows only XPath 1.0 expressions, as discussed + in . The mandatory COLUMNS clause specifies the list of columns in the output table. - If the COLUMNS clause is omitted, the rows in the result - set contain a single column of type xml containing the - data matched by row_expression. - If COLUMNS is specified, each entry describes a - single column. + Each entry describes a single column. See the syntax summary above for the format. The column name and type are required; the path, default and nullability clauses are optional. @@ -10889,42 +10888,78 @@ SELECT xpath_exists('/my:a/text()', 'test - The column_expression for a column is an XPath expression - that is evaluated for each row, relative to the result of the - row_expression, to find the value of the column. + The column_expression for a column is an XPath 1.0 + expression + that is evaluated for each row, with the current node from the + row_expression result as its context item, + to find the value of the column. If no column_expression is given, then the column name is used as an implicit path. - If a column's XPath expression returns multiple elements, an error - is raised. - If the expression matches an empty tag, the result is an - empty string (not NULL). - Any xsi:nil attributes are ignored. + If a column's XPath expression returns a non-XML value (limited to + string, boolean, or double in XPath 1.0) and the column has a + PostgreSQL type other than xml, the column will be set + as if by assigning the value's string representation to the PostgreSQL + type (adjusting the "string representation" of a boolean to + 1 or 0 if the target column type + category is numeric, otherwise true or + false). + + + + If the column's expression returns a non-empty set of XML nodes + and the target column's type is xml, the column will + be assigned the expression result exactly, if it is of document or + content form. + + + A result containing more than one element node at the top level, or + non-whitespace text outside of an element, is an example of content form. + An XPath result can be of neither form, for example if it returns an + attribute node selected from the element that contains it. Such a result + will be put into content form with each such disallowed node replaced by + its string value, as defined for the XPath 1.0 + string function. + + + + + + A non-XML result assigned to an xml output column produces + content, a single text node with the string value of the result. + An XML result assigned to a column of any other type may not have more than + one node, or an error is raised. If there is exactly one node, the column + will be set as if by assigning the node's string + value (as defined for the XPath 1.0 string function) + to the PostgreSQL type. - The text body of the XML matched by the column_expression - is used as the column value. Multiple text() nodes - within an element are concatenated in order. Any child elements, - processing instructions, and comments are ignored, but the text contents - of child elements are concatenated to the result. + The string value of an XML element is the concatenation, in document order, + of all text nodes contained in that element and its descendants. The string + value of an element with no descendant text nodes is an + empty string (not NULL). + Any xsi:nil attributes are ignored. Note that the whitespace-only text() node between two non-text elements is preserved, and that leading whitespace on a text() node is not flattened. + The XPath 1.0 string function may be consulted for the + rules defining the string value of other XML node types and non-XML values. + + + + The conversion rules presented here are not exactly those of the SQL + standard, as discussed in . - If the path expression does not match for a given row but - default_expression is specified, the value resulting - from evaluating that expression is used. - If no DEFAULT clause is given for the column, - the field will be set to NULL. - It is possible for a default_expression to reference - the value of output columns that appear prior to it in the column list, - so the default of one column may be based on the value of another - column. + If the path expression returns an empty nodeset + (typically, when it does not match) + for a given row, the column will be set to NULL, unless + a default_expression is specified; then the + value resulting from evaluating that expression is used. @@ -10936,20 +10971,14 @@ SELECT xpath_exists('/my:a/text()', 'test - Unlike regular PostgreSQL functions, column_expression - and default_expression are not evaluated to a simple - value before calling the function. - column_expression is normally evaluated - exactly once per input row, and default_expression - is evaluated each time a default is needed for a field. - If the expression qualifies as stable or immutable the repeat + A default_expression, rather than being + evaluated immediately when xmltable is called, + is evaluated each time a default is needed for the column. + If the expression qualifies as stable or immutable, the repeat evaluation may be skipped. - Effectively xmltable behaves more like a subquery than a - function call. This means that you can usefully use volatile functions like - nextval in default_expression, and - column_expression may depend on other parts of the - XML document. + nextval in + default_expression. @@ -11297,6 +11326,293 @@ table2-mapping ]]> + + + Limits and Compatibility + + + XML Functions + limits and compatibility + + + + Queries restricted to XPath 1.0 + + + The PostgreSQL-specific functions + xpath and xpath_exists query + XML documents using the XPath language, and + PostgreSQL also provides XPath-only variants of + the standard functions XMLEXISTS and + XMLTABLE, which are defined in the SQL standard to use + the XQuery language. For all of these functions, + PostgreSQL relies on the + libxml library, which provides only XPath 1.0. + + + + There is a strong connection between the XQuery language and XPath versions + 2.0 and later: any expression that is syntactically valid and executes + successfully in both produces the same result (with a minor exception for + expressions containing numeric character references or predefined entity + references, which XQuery replaces with the corresponding character while + XPath leaves them alone). But there is no such connection between XPath 1.0 + and XQuery or the later XPath versions; it was an earlier language and + differs in many respects. + + + + There are two categories of limitation to keep in mind: the restriction + from XQuery to XPath for the functions specified in the SQL standard, and + the restriction of XPath to version 1.0 for both the standard and the + PostgreSQL-specific functions. + + + + Restriction of XQuery to XPath + + + Features of XQuery beyond those of XPath include: + + + + + XQuery expressions can construct and return new XML nodes, in addition + to all possible XPath values. XPath can introduce and return values of + the atomic types (numbers, strings, and so on) but can only return XML + nodes already present in documents supplied as input to the expression. + + + + + + XQuery has control constructs for iteration, sorting, and grouping. + + + + + + XQuery allows the declaration and use of local functions. + + + + + + + Recent XPath versions begin to offer overlapping capabilities + (such as the functional-style for-each and + sort, anonymous functions, and + parse-xml to create a node from a string), + but these were not available before XPath 3.0. + + + + + Restriction of XPath to 1.0 + + + For developers familiar with XQuery and XPath 2.0 or later, or porting + queries from other systems, XPath 1.0 presents a number of differences to + contend with: + + + + + The fundamental type of an XQuery/XPath expression, the + sequence, which can contain XML nodes, atomic values, + or both, does not exist in XPath 1.0. A 1.0 expression can only produce + a nodeset (possibly empty, or with one XML node or more), or a single + atomic value. + + + + + + Unlike an XQuery/XPath sequence, which can contain any desired items + in any desired order, an XPath 1.0 nodeset has no guaranteed order and, + like any set, can have no member appear more than once. (The + libxml library does seem to always return + nodesets to PostgreSQL with their members + in the same relative order they had in the input document; it does not + commit to this behavior, and an XPath 1.0 expression cannot control + it.) + + + + + + While XQuery/XPath provides all of the types defined in XML Schema + and many operators and functions over those types, XPath 1.0 has only + nodesets and three atomic types, boolean, + double, and string. + + + + + + XPath 1.0 has no conditional operator. An XQuery/XPath expression + such as if ( hat ) then hat/@size else "no hat" + has no XPath 1.0 equivalent. + + + + + + XPath 1.0 has no ordering comparison operator for strings. Both + "cat" < "dog" and + "cat" > "dog" are false, because each is a + numeric comparison of two NaNs. In contrast, + = and != do compare the strings + as strings. + + + + + + XPath 1.0 blurs the distinction between + value comparisons and + general comparisons as XQuery/XPath define them. + Both sale/@hatsize = 7 and + sale/@customer = "alice" are existentially + quantified comparisons, true if there is any sale with the given value + for the attribute, but sale/@taxable = false() + is a value comparison to the + effective boolean value of a whole nodeset, + and true only if no sale has a taxable attribute + at all. + + + + + + In the XQuery/XPath data model, a document node + can have either document form (exactly one top-level element, only + comments and processing instructions outside of it) or content form + (with those constraints relaxed). Its equivalent in XPath 1.0, the + root node, can only be in document form. + This is part of the reason an xml value passed as the + context item to any PostgreSQL XPath-based + function must be in document form. + + + + + + + The differences highlighted here are not all of them. In XQuery and + the 2.0 and later versions of XPath, there is an XPath 1.0 compatibility + mode, and the W3C lists of function library + changes + and language + changes + applied in that mode offer a more complete account of the + differences, if still not exhaustive; the compatibility mode cannot make + the later languages exactly equivalent to XPath 1.0. + + + + + Mappings between SQL and XML data types and values + + + In SQL:2006 and later, both directions of conversion between standard SQL + data types and the XML Schema types are specified precisely. However, the + rules are expressed using the types and semantics of XQuery/XPath, and + have no direct application to the different data model of XPath 1.0. + + + + When PostgreSQL maps SQL data values to XML + (as in xmlelement), or XML to SQL (as in the output + columns of xmltable), except for the few cases + treated specially, PostgreSQL simply assumes + that the XML data type's XPath 1.0 string form will be valid as the + text-input form of the SQL datatype, and conversely. This rule has the + virtue of simplicity while producing, for many data types, results similar + to the mappings specified in the standard. + + + + Where interoperability with other systems is a concern, for some data + types, it may be necessary to use available data type formatting functions + (such as those in ) explicitly in + queries to produce the standard mappings. + + + + + + + Incidental limits of the implementation + + + + This section concerns limits that are not inherent in the + libxml library, but apply to the current + implementation in PostgreSQL. + + + + Only <literal>BY VALUE</literal> passing mechanism supported + + + The SQL standard defines two passing mechanisms + that apply when passing an XML argument from SQL to an XML function or + receiving a result: BY REF, in which a particular XML + value retains its node identity, and BY VALUE, in which + the content of the XML is passed but node identity is not preserved. A + mechanism can be specified before a list of parameters, as the default + mechanism for all of them, or after any parameter, to override the + default. + + + + To illustrate the difference, if + x is an XML value, these two queries in + an SQL:2006 environment would produce true and false, respectively: + + + + + + PostgreSQL will accept either + BY VALUE or BY REF in an + XMLEXISTS or XMLTABLE construct, + but ignores them; the xml data type holds a character-string + serialized representation, so there is no node identity to preserve, + and passing is always BY VALUE. + + + + + Cannot pass named parameters to queries + + + The XPath-based functions support passing one parameter to serve as the + XPath expression's context item, but do not support passing additional + values to be available to the expression as named parameters. + + + + + No <type>XML(SEQUENCE)</type> type + + + The PostgreSQL xml can only hold + a value in DOCUMENT or CONTENT form. + An XQuery/XPath expression context item must be a single XML node + or atomic value, while XPath 1.0 further restricts it to only an XML node, + and has no node type allowing CONTENT. The upshot is + that a well-formed DOCUMENT is the only form of XML + value that PostgreSQL can supply as an XPath + context item. + + + + diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt index bade0fe..e947613 100644 --- a/src/backend/catalog/sql_features.txt +++ b/src/backend/catalog/sql_features.txt @@ -569,7 +569,7 @@ X056 Advanced table mapping: data mapping YES X057 Advanced table mapping: metadata mapping YES X058 Advanced table mapping: base64 encoding of binary strings YES X059 Advanced table mapping: hex encoding of binary strings YES -X060 XMLParse: character string input and CONTENT option YES +X060 XMLParse: character string input and CONTENT option YES uses SQL:2003 definition, so a document with a DTD cannot be parsed as CONTENT X061 XMLParse: character string input and DOCUMENT option YES X065 XMLParse: BLOB input and CONTENT option NO X066 XMLParse: BLOB input and DOCUMENT option NO @@ -593,7 +593,7 @@ X085 Predefined namespace prefixes NO X086 XML namespace declarations in XMLTable NO X090 XML document predicate YES X091 XML content predicate NO -X096 XMLExists NO XPath only +X096 XMLExists NO XPath 1.0 only X100 Host language support for XML: CONTENT option NO X101 Host language support for XML: DOCUMENT option NO X110 Host language support for XML: VARCHAR mapping NO @@ -661,11 +661,11 @@ X282 XMLValidate with CONTENT option NO X283 XMLValidate with SEQUENCE option NO X284 XMLValidate: NAMESPACE without ELEMENT clause NO X286 XMLValidate: NO NAMESPACE with ELEMENT clause NO -X300 XMLTable NO XPath only +X300 XMLTable NO XPath 1.0 only X301 XMLTable: derived column list option YES X302 XMLTable: ordinality column option YES X303 XMLTable: column default option YES -X304 XMLTable: passing a context item YES +X304 XMLTable: passing a context item YES must be XML DOCUMENT X305 XMLTable: initializing an XQuery variable NO X400 Name and identifier mapping YES X410 Alter column data type: XML type YES --------------080405040605080502000109 Content-Type: text/x-patch; name="xml-content-2006-3.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="xml-content-2006-3.patch" diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml index 94b46a6..752eadf 100644 --- a/doc/src/sgml/datatype.sgml +++ b/doc/src/sgml/datatype.sgml @@ -4208,9 +4208,11 @@ a0ee-bc99-9c0b-4ef8-bb6d-6bb9-bd38-0a11 The xml type can store well-formed documents, as defined by the XML standard, as well - as content fragments, which are defined by the - production XMLDecl? content in the XML - standard. Roughly, this means that content fragments can have + as content fragments, which are defined by reference to the + more permissive + document node + of the XQuery and XPath data model. + Roughly, this means that content fragments can have more than one top-level element or character node. The expression xmlvalue IS DOCUMENT can be used to evaluate whether a particular xml @@ -4282,24 +4284,9 @@ SET XML OPTION { DOCUMENT | CONTENT }; SET xmloption TO { DOCUMENT | CONTENT }; The default is CONTENT, so all forms of XML - data are allowed except as noted below. + data are allowed. - - - In the SQL:2006 and later standard, the CONTENT form - is a proper superset of the DOCUMENT form, and so the - default XML option setting would allow casting to XML from character - strings in either form. PostgreSQL, however, - uses the SQL:2003 definition in which CONTENT form - cannot contain a document type declaration. Therefore, there is no one - setting of the XML option that will allow casting to XML from every valid - character string. The default will work almost always, but for a document - with a DTD, it will be necessary to change the XML option or - use XMLPARSE specifying DOCUMENT. - - - diff --git a/src/backend/catalog/sql_features.txt b/src/backend/catalog/sql_features.txt index e947613..32908c1 100644 --- a/src/backend/catalog/sql_features.txt +++ b/src/backend/catalog/sql_features.txt @@ -569,7 +569,7 @@ X056 Advanced table mapping: data mapping YES X057 Advanced table mapping: metadata mapping YES X058 Advanced table mapping: base64 encoding of binary strings YES X059 Advanced table mapping: hex encoding of binary strings YES -X060 XMLParse: character string input and CONTENT option YES uses SQL:2003 definition, so a document with a DTD cannot be parsed as CONTENT +X060 XMLParse: character string input and CONTENT option YES X061 XMLParse: character string input and DOCUMENT option YES X065 XMLParse: BLOB input and CONTENT option NO X066 XMLParse: BLOB input and DOCUMENT option NO diff --git a/src/backend/utils/adt/xml.c b/src/backend/utils/adt/xml.c index 28b3eaa..d326140 100644 --- a/src/backend/utils/adt/xml.c +++ b/src/backend/utils/adt/xml.c @@ -141,6 +141,7 @@ static int parse_xml_decl(const xmlChar *str, size_t *lenp, xmlChar **version, xmlChar **encoding, int *standalone); static bool print_xml_decl(StringInfo buf, const xmlChar *version, pg_enc encoding, int standalone); +static bool xml_doctype_in_content(const xmlChar *str); static xmlDocPtr xml_parse(text *data, XmlOptionType xmloption_arg, bool preserve_whitespace, int encoding); static text *xml_xmlnodetoxmltype(xmlNodePtr cur, PgXmlErrorContext *xmlerrcxt); @@ -1243,8 +1244,19 @@ parse_xml_decl(const xmlChar *str, size_t *lenp, if (xmlStrncmp(p, (xmlChar *) " */ - utf8len = strlen((const char *) (p + 5)); + /* + * If next char is name char, it's a PI like rather + * than an XMLDecl, so we have done what we came to do and found no XMLDecl. + * + * Use strnlen here because there is no need to go off counting all the rest + * of the input just to find out if there are enough bytes to decode one + * UTF-8 char. 6 is conservative: UTF-8 finally settled on 4 bytes as the + * longest encoding, but could some input come from a source that (a) saw + * the original drafts with up to 6 and (b) produces overlong encodings? + * Hardly likely, but it costs nearly nothing to accommodate it, while using + * 4 could be a change of behavior from when strlen was here. + */ + utf8len = strnlen((const char *) (p + 5), 6); utf8char = xmlGetUTF8Char(p + 5, &utf8len); if (PG_XMLISNAMECHAR(utf8char)) goto finished; @@ -1415,6 +1427,133 @@ print_xml_decl(StringInfo buf, const xmlChar *version, return false; } +/* + * Test whether an input that is to be parsed as CONTENT contains a DTD. + * + * The SQL/XML:2003 definition of CONTENT ("XMLDecl? content") is not satisfied + * by a document with a DTD, which is a bit of a wart, as it means the CONTENT + * type is not a proper superset of DOCUMENT. SQL/XML:2006 and later fix that, + * by redefining content with reference to the "more permissive" Document Node + * of the XQuery/XPath Data Model, such that any DOCUMENT value is indeed also a + * CONTENT value. That definition is more useful, as CONTENT becomes usable for + * parsing input of unknown form (think pg_restore). + * + * As used below in parse_xml when parsing for CONTENT, libxml does not give us + * the 2006+ behavior, but only the 2003; it will choke if the input has a DTD. + * But without the full effort of bringing our whole implementation along to + * 2006+, we can provide the 2006+ definition of CONTENT easily enough, by + * detecting this case first and simply doing the parse as DOCUMENT. + * + * A DTD can be found arbitrarily far in, but that would be a contrived case; it + * will ordinarily start within a few dozen characters. We only need to see the + * start. The only things that can precede it are an XMLDecl (here, the caller + * will have called parse_xml_decl already), whitespace, comments, and + * processing instructions. This function need only return true if it sees a + * valid sequence of such things leading to must follow */ + p = xmlStrstr(p + 2, (xmlChar *) "--"); + if ( !p || '>' != p[2] ) + return false; /* didn't find -- or > didn't follow */ + p += 3; /* point to char following --> */ + continue; + } + + if ( '?' != *p ) /* a PI is the only choice left */ + return false; + + ++ p; /* point to char following is forbidden within a PI; it can only mean the end. */ + e = xmlStrstr(p, (xmlChar *) "?>"); + if ( !e ) + return false; + + /* + * The target is from p to the first blank (or end of PI if no blank). + * The characters accepted by xmlIsBlank are all 7-bit; it will not be + * confounded by any multibyte UTF-8 characters, though it will end up + * seeing each byte individually. + */ + for ( sp = p; sp < e; ++ sp ) + if ( xmlIsBlank_ch(*sp) ) + break; + + if ( sp == p ) + return false; /* there should have been at least one non-blank */ + + /* The target must not be, case-insensitively, "xml". */ + if ( sp - p == 3 && 0 == xmlStrncasecmp(p, (xmlChar *) "xml", 3) ) + return false; + + utf8len = sp - p; + utf8char = xmlGetUTF8Char(p, &utf8len); + if ( ! PG_XMLISNAMECHAR(utf8char) ) + return false; + /* certain chars are excluded as the FIRST name char */ + if ( '-' == utf8char || '.' == utf8char || 0xB7 == utf8char || + ('0' <= utf8char && utf8char <= '9') || + ( 0x300 <= utf8char && utf8char <= 0x36F ) || + 0x203F == utf8char || 0x2040 == utf8char ) + return false; + + p += utf8len; + + while ( p < sp ) /* the rest can be anything PG_XMLISNAMECHAR accepts */ + { + utf8len = sp - p; + utf8char = xmlGetUTF8Char(p, &utf8len); + if ( ! PG_XMLISNAMECHAR(utf8char) ) + return false; + p += utf8len; + } + + /* + * The target checks out as a NAME. There are no constraints on the rest + * of the content in a PI (besides that it can't be ?> relied on above), + * so it's good. Point at the char following the PI's ending ?> + */ + p = e + 2; + } +} + /* * Convert a C string to XML internal representation @@ -1433,6 +1572,9 @@ xml_parse(text *data, XmlOptionType xmloption_arg, bool preserve_whitespace, xmlChar *string; xmlChar *utf8string; PgXmlErrorContext *xmlerrcxt; + int errcode_document_parse = ERRCODE_INVALID_XML_DOCUMENT; + char const *errmsg_document_parse = "invalid XML document"; + char const *errmsg_content_parse = "invalid XML content"; volatile xmlParserCtxtPtr ctxt = NULL; volatile xmlDocPtr doc = NULL; @@ -1457,6 +1599,7 @@ xml_parse(text *data, XmlOptionType xmloption_arg, bool preserve_whitespace, xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, "could not allocate parser context"); +try_other_xmloption: if (xmloption_arg == XMLOPTION_DOCUMENT) { /* @@ -1472,8 +1615,8 @@ xml_parse(text *data, XmlOptionType xmloption_arg, bool preserve_whitespace, XML_PARSE_NOENT | XML_PARSE_DTDATTR | (preserve_whitespace ? 0 : XML_PARSE_NOBLANKS)); if (doc == NULL || xmlerrcxt->err_occurred) - xml_ereport(xmlerrcxt, ERROR, ERRCODE_INVALID_XML_DOCUMENT, - "invalid XML document"); + xml_ereport(xmlerrcxt, ERROR, errcode_document_parse, + errmsg_document_parse); } else { @@ -1489,6 +1632,21 @@ xml_parse(text *data, XmlOptionType xmloption_arg, bool preserve_whitespace, "invalid XML content: invalid XML declaration", res_code); + /* + * The way we use libxml below to parse CONTENT will not accept the + * case (not allowed in SQL/XML:2003 but allowed in 2006+) where the + * content is, in fact, a document with a DTD. By detecting that + * case here and parsing as DOCUMENT instead, we provide the 2006+ + * behavior. + */ + if (xml_doctype_in_content(utf8string + count)) + { + xmloption_arg = XMLOPTION_DOCUMENT; + errcode_document_parse = ERRCODE_INVALID_XML_CONTENT; + errmsg_document_parse = errmsg_content_parse; + goto try_other_xmloption; + } + doc = xmlNewDoc(version); Assert(doc->encoding == NULL); doc->encoding = xmlStrdup((const xmlChar *) "UTF-8"); @@ -1501,7 +1659,7 @@ xml_parse(text *data, XmlOptionType xmloption_arg, bool preserve_whitespace, utf8string + count, NULL); if (res_code != 0 || xmlerrcxt->err_occurred) xml_ereport(xmlerrcxt, ERROR, ERRCODE_INVALID_XML_CONTENT, - "invalid XML content"); + errmsg_content_parse); } } } diff --git a/src/test/regress/expected/xml.out b/src/test/regress/expected/xml.out index 2085fa0..0aae600 100644 --- a/src/test/regress/expected/xml.out +++ b/src/test/regress/expected/xml.out @@ -532,6 +532,13 @@ LINE 1: EXECUTE foo ('bad'); DETAIL: line 1: Start tag expected, '<' not found bad ^ +SELECT xml ''; +ERROR: invalid XML document +LINE 1: SELECT xml ''; + ^ +DETAIL: line 1: Extra content at the end of the document + + ^ SET XML OPTION CONTENT; EXECUTE foo (''); xmlconcat @@ -545,6 +552,45 @@ EXECUTE foo ('good'); good (1 row) +SELECT xml ' '; + xml +-------------------------------------------------------------------- + +(1 row) + +SELECT xml ' '; + xml +------------------------------ + +(1 row) + +SELECT xml ''; + xml +------------------ + +(1 row) + +SELECT xml ' oops '; +ERROR: invalid XML content +LINE 1: SELECT xml ' oops '; + ^ +DETAIL: line 1: StartTag: invalid element name + oops + ^ +SELECT xml ' '; +ERROR: invalid XML content +LINE 1: SELECT xml ' '; + ^ +DETAIL: line 1: StartTag: invalid element name + + ^ +SELECT xml ''; +ERROR: invalid XML content +LINE 1: SELECT xml ''; + ^ +DETAIL: line 1: Extra content at the end of the document + + ^ -- Test backwards parsing CREATE VIEW xmlview1 AS SELECT xmlcomment('test'); CREATE VIEW xmlview2 AS SELECT xmlconcat('hello', 'you'); diff --git a/src/test/regress/expected/xml_1.out b/src/test/regress/expected/xml_1.out index 7f86696..d1a03b5 100644 --- a/src/test/regress/expected/xml_1.out +++ b/src/test/regress/expected/xml_1.out @@ -429,11 +429,53 @@ EXECUTE foo (''); ERROR: prepared statement "foo" does not exist EXECUTE foo ('bad'); ERROR: prepared statement "foo" does not exist +SELECT xml ''; +ERROR: unsupported XML feature +LINE 1: SELECT xml ''; + ^ +DETAIL: This functionality requires the server to be built with libxml support. +HINT: You need to rebuild PostgreSQL using --with-libxml. SET XML OPTION CONTENT; EXECUTE foo (''); ERROR: prepared statement "foo" does not exist EXECUTE foo ('good'); ERROR: prepared statement "foo" does not exist +SELECT xml ' '; +ERROR: unsupported XML feature +LINE 1: SELECT xml ' '; +ERROR: unsupported XML feature +LINE 1: SELECT xml ' '; +ERROR: unsupported XML feature +LINE 1: SELECT xml ''; + ^ +DETAIL: This functionality requires the server to be built with libxml support. +HINT: You need to rebuild PostgreSQL using --with-libxml. +SELECT xml ' oops '; +ERROR: unsupported XML feature +LINE 1: SELECT xml ' oops '; + ^ +DETAIL: This functionality requires the server to be built with libxml support. +HINT: You need to rebuild PostgreSQL using --with-libxml. +SELECT xml ' '; +ERROR: unsupported XML feature +LINE 1: SELECT xml ' '; + ^ +DETAIL: This functionality requires the server to be built with libxml support. +HINT: You need to rebuild PostgreSQL using --with-libxml. +SELECT xml ''; +ERROR: unsupported XML feature +LINE 1: SELECT xml ''; + ^ +DETAIL: This functionality requires the server to be built with libxml support. +HINT: You need to rebuild PostgreSQL using --with-libxml. -- Test backwards parsing CREATE VIEW xmlview1 AS SELECT xmlcomment('test'); CREATE VIEW xmlview2 AS SELECT xmlconcat('hello', 'you'); diff --git a/src/test/regress/expected/xml_2.out b/src/test/regress/expected/xml_2.out index 510b09b..9756b6e 100644 --- a/src/test/regress/expected/xml_2.out +++ b/src/test/regress/expected/xml_2.out @@ -512,6 +512,13 @@ LINE 1: EXECUTE foo ('bad'); DETAIL: line 1: Start tag expected, '<' not found bad ^ +SELECT xml ''; +ERROR: invalid XML document +LINE 1: SELECT xml ''; + ^ +DETAIL: line 1: Extra content at the end of the document + + ^ SET XML OPTION CONTENT; EXECUTE foo (''); xmlconcat @@ -525,6 +532,45 @@ EXECUTE foo ('good'); good (1 row) +SELECT xml ' '; + xml +-------------------------------------------------------------------- + +(1 row) + +SELECT xml ' '; + xml +------------------------------ + +(1 row) + +SELECT xml ''; + xml +------------------ + +(1 row) + +SELECT xml ' oops '; +ERROR: invalid XML content +LINE 1: SELECT xml ' oops '; + ^ +DETAIL: line 1: StartTag: invalid element name + oops + ^ +SELECT xml ' '; +ERROR: invalid XML content +LINE 1: SELECT xml ' '; + ^ +DETAIL: line 1: StartTag: invalid element name + + ^ +SELECT xml ''; +ERROR: invalid XML content +LINE 1: SELECT xml ''; + ^ +DETAIL: line 1: Extra content at the end of the document + + ^ -- Test backwards parsing CREATE VIEW xmlview1 AS SELECT xmlcomment('test'); CREATE VIEW xmlview2 AS SELECT xmlconcat('hello', 'you'); diff --git a/src/test/regress/sql/xml.sql b/src/test/regress/sql/xml.sql index 8057a46..71431d8 100644 --- a/src/test/regress/sql/xml.sql +++ b/src/test/regress/sql/xml.sql @@ -149,10 +149,17 @@ PREPARE foo (xml) AS SELECT xmlconcat('', $1); SET XML OPTION DOCUMENT; EXECUTE foo (''); EXECUTE foo ('bad'); +SELECT xml ''; SET XML OPTION CONTENT; EXECUTE foo (''); EXECUTE foo ('good'); +SELECT xml ' '; +SELECT xml ' '; +SELECT xml ''; +SELECT xml ' oops '; +SELECT xml ' '; +SELECT xml ''; -- Test backwards parsing --------------080405040605080502000109--