public inbox for [email protected]help / color / mirror / Atom feed
Re: [PATCH] Add CANONICAL option to xmlserialize 28+ messages / 3 participants [nested] [flat]
* Re: [PATCH] Add CANONICAL option to xmlserialize @ 2023-03-05 21:00 Thomas Munro <[email protected]> 2023-03-05 22:20 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 0 siblings, 1 reply; 28+ messages in thread From: Thomas Munro @ 2023-03-05 21:00 UTC (permalink / raw) To: Jim Jones <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> On Mon, Mar 6, 2023 at 7:44 AM Jim Jones <[email protected]> wrote: > The attached version includes documentation and tests to the patch. The CI run for that failed in an interesting way, only on Debian + Meson, 32 bit. The diffs appear to show that psql has a different opinion of the column width, while building its header (the "------" you get at the top of psql's output), even though the actual column contents was the same. regression.diff[2] shows that there is a "£1" in the output, which is how UTF-8 "£1" looks if you view it with Latin1 glasses on. Clearly this patch involves transcoding, Latin1 and UTF-8 and I haven't studied it, but it's pretty weird for the 32 bit build to give a different result... could be something to do with our environment, since .cirrus.yml sets LANG=C in the 32 bit test run -- maybe try that locally? That run also generated a core dump, but I think that's just our open SIGQUIT problem[3] and not relevant here. [1] https://cirrus-ci.com/build/6319462375227392 [2] https://api.cirrus-ci.com/v1/artifact/task/5800598633709568/testrun/build-32/testrun/regress/regress... [3] https://www.postgresql.org/message-id/flat/20230214202927.xgb2w6b7gnhq6tvv%40awork3.anarazel.de ^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: [PATCH] Add CANONICAL option to xmlserialize 2023-03-05 21:00 Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> @ 2023-03-05 22:20 ` Jim Jones <[email protected]> 2023-03-05 23:32 ` Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 0 siblings, 1 reply; 28+ messages in thread From: Jim Jones @ 2023-03-05 22:20 UTC (permalink / raw) To: Thomas Munro <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> On 05.03.23 22:00, Thomas Munro wrote: > The CI run for that failed in an interesting way, only on Debian + > Meson, 32 bit. The diffs appear to show that psql has a different > opinion of the column width, while building its header (the "------" > you get at the top of psql's output), even though the actual column > contents was the same. regression.diff[2] shows that there is a "£1" > in the output, which is how UTF-8 "£1" looks if you view it with > Latin1 glasses on. Clearly this patch involves transcoding, Latin1 > and UTF-8 One of the use cases of this patch is exactly the transcoding of a non utf-8 document to utf-8 - as described in the XML canonical spec. > and I haven't studied it, but it's pretty weird for the 32 > bit build to give a different result... Yeah, it's pretty weird indeed. I'll try to reproduce this environment in a container to see if I get the same diff. Although I'm not sure that by "fixing" the result set for this environment it won't break all the others. > could be something to do with > our environment, since .cirrus.yml sets LANG=C in the 32 bit test run > -- maybe try that locally? Also using LANGUAGE=C the result is the same for me - all tests pass just fine. > That run also generated a core dump, but I think that's just our open > SIGQUIT problem[3] and not relevant here. > > [1] https://cirrus-ci.com/build/6319462375227392 > [2] https://api.cirrus-ci.com/v1/artifact/task/5800598633709568/testrun/build-32/testrun/regress/regress... > [3] https://www.postgresql.org/message-id/flat/20230214202927.xgb2w6b7gnhq6tvv%40awork3.anarazel.de Thanks for the quick reply. Much appreciated! ^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: [PATCH] Add CANONICAL option to xmlserialize 2023-03-05 21:00 Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-05 22:20 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> @ 2023-03-05 23:32 ` Thomas Munro <[email protected]> 2023-03-06 10:50 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 0 siblings, 1 reply; 28+ messages in thread From: Thomas Munro @ 2023-03-05 23:32 UTC (permalink / raw) To: Jim Jones <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> On Mon, Mar 6, 2023 at 11:20 AM Jim Jones <[email protected]> wrote: > On 05.03.23 22:00, Thomas Munro wrote: > > could be something to do with > > our environment, since .cirrus.yml sets LANG=C in the 32 bit test run > > -- maybe try that locally? > Also using LANGUAGE=C the result is the same for me - all tests pass > just fine. I couldn't reproduce that locally either, but I just tested on CI with your patch applied saw the failure, and then removed "PYTHONCOERCECLOCALE=0 LANG=C" and it's all green: https://github.com/macdice/postgres/commit/91999f5d13ac2df6f7237a301ed6cf73f2bb5b6d Without looking too closely, my first guess would have been that this just isn't going to work without UTF-8 database encoding, so you might need to skip the test (see for example src/test/regress/expected/unicode_1.out). It's annoying that "xml" already has 3 expected variants... hmm. BTW shouldn't it be failing in a more explicit way somewhere sooner if the database encoding is not UTF-8, rather than getting confused? ^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: [PATCH] Add CANONICAL option to xmlserialize 2023-03-05 21:00 Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-05 22:20 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-05 23:32 ` Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> @ 2023-03-06 10:50 ` Jim Jones <[email protected]> 2023-03-06 13:19 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 0 siblings, 1 reply; 28+ messages in thread From: Jim Jones @ 2023-03-06 10:50 UTC (permalink / raw) To: Thomas Munro <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> On 06.03.23 00:32, Thomas Munro wrote: > I couldn't reproduce that locally either, but I just tested on CI with > your patch applied saw the failure, and then removed > "PYTHONCOERCECLOCALE=0 LANG=C" and it's all green: > > https://github.com/macdice/postgres/commit/91999f5d13ac2df6f7237a301ed6cf73f2bb5b6d > > Without looking too closely, my first guess would have been that this > just isn't going to work without UTF-8 database encoding, so you might > need to skip the test (see for example > src/test/regress/expected/unicode_1.out). It's annoying that "xml" > already has 3 expected variants... hmm. BTW shouldn't it be failing > in a more explicit way somewhere sooner if the database encoding is > not UTF-8, rather than getting confused? I guess this confusion is happening because xml_parse() was being called with the database encoding from GetDatabaseEncoding(). I added a condition before calling xml_parse() to check if the xml document has a different encoding than UTF-8 parse_xml_decl(xml_text2xmlChar(data), NULL, NULL, &encodingStr, NULL); encoding = encodingStr ? xmlChar_to_encoding(encodingStr) : PG_UTF8; doc = xml_parse(data, XMLOPTION_DOCUMENT, false, encoding, NULL); v2 attached. Thanks! Best, Jim Attachments: [text/x-patch] v2-0001-Add-CANONICAL-format-to-xmlserialize.patch (32.0K, ../../[email protected]/2-v2-0001-Add-CANONICAL-format-to-xmlserialize.patch) download | inline diff: From 23ed616809b319bb175473bd210d81ad95af21ff Mon Sep 17 00:00:00 2001 From: Jim Jones <[email protected]> Date: Tue, 28 Feb 2023 23:06:30 +0100 Subject: [PATCH v2] Add CANONICAL format to xmlserialize This patch introduces the CANONICAL option to xmlserialize, which serializes xml documents in their canonical form - as described in the W3C Canonical XML Version 1.1 specification. This option can be used with the additional parameter WITH [NO] COMMENTS to keep or remove xml comments from the canonical xml output. This feature is based on the function xmlC14NDocDumpMemory from the C14N module of libxml2. This patch also includes regression tests and documentation. --- doc/src/sgml/datatype.sgml | 41 +++++++++- src/backend/executor/execExprInterp.c | 12 ++- src/backend/parser/gram.y | 14 +++- src/backend/parser/parse_expr.c | 1 + src/backend/utils/adt/xml.c | 70 +++++++++++++++++ src/include/nodes/parsenodes.h | 1 + src/include/nodes/primnodes.h | 9 +++ src/include/parser/kwlist.h | 1 + src/include/utils/xml.h | 1 + src/test/regress/expected/xml.out | 108 ++++++++++++++++++++++++++ src/test/regress/expected/xml_1.out | 104 +++++++++++++++++++++++++ src/test/regress/expected/xml_2.out | 108 ++++++++++++++++++++++++++ src/test/regress/sql/xml.sql | 61 +++++++++++++++ 13 files changed, 526 insertions(+), 5 deletions(-) diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml index 467b49b199..46ec95dbb8 100644 --- a/doc/src/sgml/datatype.sgml +++ b/doc/src/sgml/datatype.sgml @@ -4460,7 +4460,7 @@ xml '<foo>bar</foo>' <type>xml</type>, uses the function <function>xmlserialize</function>:<indexterm><primary>xmlserialize</primary></indexterm> <synopsis> -XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <replaceable>type</replaceable> ) +XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <replaceable>type</replaceable> [ CANONICAL [ WITH [NO] COMMENTS ] ]) </synopsis> <replaceable>type</replaceable> can be <type>character</type>, <type>character varying</type>, or @@ -4470,6 +4470,45 @@ XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <repla you to simply cast the value. </para> + <para> + The option <type>CANONICAL</type> converts a given + XML document to its <ulink url="https://www.w3.org/TR/xml-c14n11/#Terminology">canonical form</ulink> + based on the <ulink url="https://www.w3.org/TR/xml-c14n11/">W3C Canonical XML 1.1 Specification</ulink>. + It is basically designed to provide applications the ability to compare xml documents or test if they + have been changed. The optional parameter <type>WITH [NO] COMMENTS</type> removes or keeps XML comments + from the given document. + </para> + + <para> + Example: + +<screen><![CDATA[ +SELECT + xmlserialize(DOCUMENT + '<foo> + <!-- a comment --> + <bar c="3" b="2" a="1">42</bar> + <empty/> + </foo>'::xml AS text CANONICAL); + xmlserialize +----------------------------------------------------------- + <foo><bar a="1" b="2" c="3">42</bar><empty></empty></foo> +(1 row) + +SELECT + xmlserialize(DOCUMENT + '<foo> + <!-- a comment --> + <bar c="3" b="2" a="1">42</bar> + <empty/> + </foo>'::xml AS text CANONICAL WITH COMMENTS); + xmlserialize +----------------------------------------------------------------------------- + <foo><!-- a comment --><bar a="1" b="2" c="3">42</bar><empty></empty></foo> +(1 row) + +]]></screen> + </para> <para> When a character string value is cast to or from type <type>xml</type> without going through <type>XMLPARSE</type> or diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c index 19351fe34b..f8f10f0ed9 100644 --- a/src/backend/executor/execExprInterp.c +++ b/src/backend/executor/execExprInterp.c @@ -3829,6 +3829,8 @@ ExecEvalXmlExpr(ExprState *state, ExprEvalStep *op) { Datum *argvalue = op->d.xmlexpr.argvalue; bool *argnull = op->d.xmlexpr.argnull; + XmlSerializeFormat format = op->d.xmlexpr.xexpr->format; + text *data; /* argument type is known to be xml */ Assert(list_length(xexpr->args) == 1); @@ -3837,9 +3839,15 @@ ExecEvalXmlExpr(ExprState *state, ExprEvalStep *op) return; value = argvalue[0]; - *op->resvalue = PointerGetDatum(xmltotext_with_xmloption(DatumGetXmlP(value), - xexpr->xmloption)); *op->resnull = false; + + data = xmltotext_with_xmloption(DatumGetXmlP(value), + xexpr->xmloption); + + if (format == XMLDEFAULT_FORMAT) + *op->resvalue = PointerGetDatum(data); + else if (format == XMLCANONICAL || format == XMLCANONICAL_WITH_COMMENTS) + *op->resvalue = PointerGetDatum(xmlserialize_canonical(data,format)); } break; diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index a0138382a1..af5f3dfdfd 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -619,6 +619,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <defelt> xmltable_column_option_el %type <list> xml_namespace_list %type <target> xml_namespace_el +%type <ival> opt_xml_serialize_format %type <node> func_application func_expr_common_subexpr %type <node> func_expr func_expr_windowless @@ -676,7 +677,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); BACKWARD BEFORE BEGIN_P BETWEEN BIGINT BINARY BIT BOOLEAN_P BOTH BREADTH BY - CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P + CACHE CALL CALLED CANONICAL CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT @@ -15532,13 +15533,14 @@ func_expr_common_subexpr: $$ = makeXmlExpr(IS_XMLROOT, NULL, NIL, list_make3($3, $5, $6), @1); } - | XMLSERIALIZE '(' document_or_content a_expr AS SimpleTypename ')' + | XMLSERIALIZE '(' document_or_content a_expr AS SimpleTypename opt_xml_serialize_format ')' { XmlSerialize *n = makeNode(XmlSerialize); n->xmloption = $3; n->expr = $4; n->typeName = $6; + n->format = $7; n->location = @1; $$ = (Node *) n; } @@ -15622,6 +15624,12 @@ xml_passing_mech: | BY VALUE_P ; +opt_xml_serialize_format: + CANONICAL { $$ = XMLCANONICAL; } + | CANONICAL WITH NO COMMENTS { $$ = XMLCANONICAL; } + | CANONICAL WITH COMMENTS { $$ = XMLCANONICAL_WITH_COMMENTS; } + | /*EMPTY*/ { $$ = XMLDEFAULT_FORMAT; } + ; /* * Aggregate decoration clauses @@ -16737,6 +16745,7 @@ unreserved_keyword: | CACHE | CALL | CALLED + | CANONICAL | CASCADE | CASCADED | CATALOG_P @@ -17259,6 +17268,7 @@ bare_label_keyword: | CACHE | CALL | CALLED + | CANONICAL | CASCADE | CASCADED | CASE diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 7ff41acb84..ddfbfe259d 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -2332,6 +2332,7 @@ transformXmlSerialize(ParseState *pstate, XmlSerialize *xs) xexpr->xmloption = xs->xmloption; xexpr->location = xs->location; + xexpr->format = xs->format; /* We actually only need these to be able to parse back the expression. */ xexpr->type = targetType; xexpr->typmod = targetTypmod; diff --git a/src/backend/utils/adt/xml.c b/src/backend/utils/adt/xml.c index 079bcb1208..f618be524b 100644 --- a/src/backend/utils/adt/xml.c +++ b/src/backend/utils/adt/xml.c @@ -56,6 +56,7 @@ #include <libxml/xmlwriter.h> #include <libxml/xpath.h> #include <libxml/xpathInternals.h> +#include <libxml/c14n.h> /* * We used to check for xmlStructuredErrorContext via a configure test; but @@ -4818,3 +4819,72 @@ XmlTableDestroyOpaque(TableFuncScanState *state) NO_XML_SUPPORT(); #endif /* not USE_LIBXML */ } + +xmltype * +xmlserialize_canonical(text *data, XmlSerializeFormat format) +{ +#ifdef USE_LIBXML + + xmlDocPtr doc; + xmlChar *xmlbuf = NULL; + xmlChar *encodingStr = NULL; + int nbytes; + int with_comments = 0; /* 0 = remove xml comments (default) */ + int encoding; + StringInfoData buf; + + + if (format != XMLCANONICAL && format != XMLCANONICAL_WITH_COMMENTS) + elog(ERROR,"invalid canonical xml option"); + else if (format == XMLCANONICAL_WITH_COMMENTS) + with_comments = 1; + + /* + * Parses the encoding from the header, so that xml_parse() knows what + * it is dealing with. If not explicitly declared, we assume UTF-8. + */ + parse_xml_decl(xml_text2xmlChar(data), NULL, NULL, &encodingStr, NULL); + encoding = encodingStr ? xmlChar_to_encoding(encodingStr) : PG_UTF8; + + doc = xml_parse(data, XMLOPTION_DOCUMENT, false, encoding, NULL); + + if(!doc) + elog(ERROR, "could not parse the given XML document"); + + /* + * int + * xmlC14NDocDumpMemory ( + * xmlDocPtr doc, # the XML document for canonization + * xmlNodeSetPtr nodes, # the nodes set to be included in the canonized image + * or NULL if all document nodes should be included + * int mode, # 0 = Original C14N 1.0 (Outdated) + * 1 = Exclusive C14N 1.0 (Outdated) + * 2 = C14N 1.1 + * xmlChar **inclusive_ns_prefixes, # the list of inclusive namespace prefixes ended with + * a NULL or NULL if there is no inclusive namespaces + * (only for exclusive canonicalization, ignored otherwise) + * int with_comments, # include comments in the result (!=0) or not (==0) + * xmlChar **xmlbuf # the memory pointer for allocated canonical XML text; + * ) + * Returns: the number of bytes written on success or a negative value on fail. + */ + + nbytes = xmlC14NDocDumpMemory(doc, NULL, 2, NULL, with_comments, &xmlbuf); + + xmlFreeDoc(doc); + + if(nbytes < 0) + elog(ERROR,"could not canonicalize the given XML document"); + + initStringInfo(&buf); + appendStringInfoString(&buf, (const char *) xmlbuf); + + xmlFree(xmlbuf); + + return stringinfo_to_xmltype(&buf); + +#else + NO_XML_SUPPORT(); + return 0; +#endif +} \ No newline at end of file diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index f7d7f10f7d..8ba0984266 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -842,6 +842,7 @@ typedef struct XmlSerialize Node *expr; TypeName *typeName; int location; /* token location, or -1 if unknown */ + XmlSerializeFormat format; /* serialization format */ } XmlSerialize; /* Partitioning related definitions */ diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h index b4292253cc..79fdcc2650 100644 --- a/src/include/nodes/primnodes.h +++ b/src/include/nodes/primnodes.h @@ -1471,6 +1471,13 @@ typedef enum XmlOptionType XMLOPTION_CONTENT } XmlOptionType; +typedef enum XmlSerializeFormat +{ + XMLCANONICAL, /* canonical form without xml comments */ + XMLCANONICAL_WITH_COMMENTS, /* canonical form with xml comments */ + XMLDEFAULT_FORMAT /* unformatted xml representation */ +} XmlSerializeFormat; + typedef struct XmlExpr { Expr xpr; @@ -1491,6 +1498,8 @@ typedef struct XmlExpr int32 typmod pg_node_attr(query_jumble_ignore); /* token location, or -1 if unknown */ int location; + /* serialization format: XMLCANONICAL, XMLCANONICAL_WITH_COMMENTS, XMLINDENT */ + XmlSerializeFormat format pg_node_attr(query_jumble_ignore); } XmlExpr; /* ---------------- diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index bb36213e6f..c1b1a720fe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -67,6 +67,7 @@ PG_KEYWORD("by", BY, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cache", CACHE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("call", CALL, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("called", CALLED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("canonical", CANONICAL, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cascade", CASCADE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cascaded", CASCADED, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("case", CASE, RESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/utils/xml.h b/src/include/utils/xml.h index 311da06cd6..745ebefe24 100644 --- a/src/include/utils/xml.h +++ b/src/include/utils/xml.h @@ -90,4 +90,5 @@ extern PGDLLIMPORT int xmloption; /* XmlOptionType, but int for guc enum */ extern PGDLLIMPORT const TableFuncRoutine XmlTableRoutine; +xmltype *xmlserialize_canonical(text *data, XmlSerializeFormat format); #endif /* XML_H */ diff --git a/src/test/regress/expected/xml.out b/src/test/regress/expected/xml.out index 3c357a9c7e..de3bfabcef 100644 --- a/src/test/regress/expected/xml.out +++ b/src/test/regress/expected/xml.out @@ -486,6 +486,114 @@ SELECT xmlserialize(content 'good' as char(10)); SELECT xmlserialize(document 'bad' as text); ERROR: not an XML document +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>£1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>£1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>£1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ?column? ---------- diff --git a/src/test/regress/expected/xml_1.out b/src/test/regress/expected/xml_1.out index 378b412db0..8b5be34c50 100644 --- a/src/test/regress/expected/xml_1.out +++ b/src/test/regress/expected/xml_1.out @@ -309,6 +309,110 @@ ERROR: unsupported XML feature LINE 1: SELECT xmlserialize(document 'bad' as text); ^ DETAIL: This functionality requires the server to be built with libxml support. +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>£1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +ERROR: unsupported XML feature +LINE 2: (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + ^ +DETAIL: This functionality requires the server to be built with libxml support. +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- +(0 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ERROR: unsupported XML feature LINE 1: SELECT xml '<foo>bar</foo>' IS DOCUMENT; diff --git a/src/test/regress/expected/xml_2.out b/src/test/regress/expected/xml_2.out index 42055c5003..9feeb9301e 100644 --- a/src/test/regress/expected/xml_2.out +++ b/src/test/regress/expected/xml_2.out @@ -466,6 +466,114 @@ SELECT xmlserialize(content 'good' as char(10)); SELECT xmlserialize(document 'bad' as text); ERROR: not an XML document +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>£1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>£1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>£1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ?column? ---------- diff --git a/src/test/regress/sql/xml.sql b/src/test/regress/sql/xml.sql index ddff459297..32bc650a9d 100644 --- a/src/test/regress/sql/xml.sql +++ b/src/test/regress/sql/xml.sql @@ -132,6 +132,67 @@ SELECT xmlserialize(content data as character varying(20)) FROM xmltest; SELECT xmlserialize(content 'good' as char(10)); SELECT xmlserialize(document 'bad' as text); +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>£1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; SELECT xml '<foo>bar</foo><bar>foo</bar>' IS DOCUMENT; -- 2.25.1 ^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: [PATCH] Add CANONICAL option to xmlserialize 2023-03-05 21:00 Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-05 22:20 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-05 23:32 ` Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-06 10:50 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> @ 2023-03-06 13:19 ` Jim Jones <[email protected]> 2023-03-14 07:49 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 0 siblings, 1 reply; 28+ messages in thread From: Jim Jones @ 2023-03-06 13:19 UTC (permalink / raw) To: Thomas Munro <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> On 06.03.23 11:50, I wrote: > I guess this confusion is happening because xml_parse() was being > called with the database encoding from GetDatabaseEncoding(). > > I added a condition before calling xml_parse() to check if the xml > document has a different encoding than UTF-8 > > parse_xml_decl(xml_text2xmlChar(data), NULL, NULL, &encodingStr, NULL); > encoding = encodingStr ? xmlChar_to_encoding(encodingStr) : PG_UTF8; > > doc = xml_parse(data, XMLOPTION_DOCUMENT, false, encoding, NULL); It seems that this bug fix didn't change the output of the CI on Debian + Meson, 32bit. I slightly changed the test case to a character that both encodings can deal with. v3 attached. Attachments: [text/x-patch] v3-0001-Add-CANONICAL-format-to-xmlserialize.patch (31.9K, ../../[email protected]/2-v3-0001-Add-CANONICAL-format-to-xmlserialize.patch) download | inline diff: From 759bacecb7e9355f41ebd02cbca38cf893e3cfc0 Mon Sep 17 00:00:00 2001 From: Jim Jones <[email protected]> Date: Mon, 6 Mar 2023 14:08:54 +0100 Subject: [PATCH v3] Add CANONICAL format to xmlserialize This patch introduces the CANONICAL option to xmlserialize, which serializes xml documents in their canonical form - as described in the W3C Canonical XML Version 1.1 specification. This option can be used with the additional parameter WITH [NO] COMMENTS to keep or remove xml comments from the canonical xml output. This feature is based on the function xmlC14NDocDumpMemory from the C14N module of libxml2. This patch also includes regression tests and documentation. --- doc/src/sgml/datatype.sgml | 41 +++++++++- src/backend/executor/execExprInterp.c | 12 ++- src/backend/parser/gram.y | 14 +++- src/backend/parser/parse_expr.c | 1 + src/backend/utils/adt/xml.c | 70 +++++++++++++++++ src/include/nodes/parsenodes.h | 1 + src/include/nodes/primnodes.h | 9 +++ src/include/parser/kwlist.h | 1 + src/include/utils/xml.h | 1 + src/test/regress/expected/xml.out | 108 ++++++++++++++++++++++++++ src/test/regress/expected/xml_1.out | 104 +++++++++++++++++++++++++ src/test/regress/expected/xml_2.out | 108 ++++++++++++++++++++++++++ src/test/regress/sql/xml.sql | 61 +++++++++++++++ 13 files changed, 526 insertions(+), 5 deletions(-) diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml index 467b49b199..46ec95dbb8 100644 --- a/doc/src/sgml/datatype.sgml +++ b/doc/src/sgml/datatype.sgml @@ -4460,7 +4460,7 @@ xml '<foo>bar</foo>' <type>xml</type>, uses the function <function>xmlserialize</function>:<indexterm><primary>xmlserialize</primary></indexterm> <synopsis> -XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <replaceable>type</replaceable> ) +XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <replaceable>type</replaceable> [ CANONICAL [ WITH [NO] COMMENTS ] ]) </synopsis> <replaceable>type</replaceable> can be <type>character</type>, <type>character varying</type>, or @@ -4470,6 +4470,45 @@ XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <repla you to simply cast the value. </para> + <para> + The option <type>CANONICAL</type> converts a given + XML document to its <ulink url="https://www.w3.org/TR/xml-c14n11/#Terminology">canonical form</ulink> + based on the <ulink url="https://www.w3.org/TR/xml-c14n11/">W3C Canonical XML 1.1 Specification</ulink>. + It is basically designed to provide applications the ability to compare xml documents or test if they + have been changed. The optional parameter <type>WITH [NO] COMMENTS</type> removes or keeps XML comments + from the given document. + </para> + + <para> + Example: + +<screen><![CDATA[ +SELECT + xmlserialize(DOCUMENT + '<foo> + <!-- a comment --> + <bar c="3" b="2" a="1">42</bar> + <empty/> + </foo>'::xml AS text CANONICAL); + xmlserialize +----------------------------------------------------------- + <foo><bar a="1" b="2" c="3">42</bar><empty></empty></foo> +(1 row) + +SELECT + xmlserialize(DOCUMENT + '<foo> + <!-- a comment --> + <bar c="3" b="2" a="1">42</bar> + <empty/> + </foo>'::xml AS text CANONICAL WITH COMMENTS); + xmlserialize +----------------------------------------------------------------------------- + <foo><!-- a comment --><bar a="1" b="2" c="3">42</bar><empty></empty></foo> +(1 row) + +]]></screen> + </para> <para> When a character string value is cast to or from type <type>xml</type> without going through <type>XMLPARSE</type> or diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c index 19351fe34b..f8f10f0ed9 100644 --- a/src/backend/executor/execExprInterp.c +++ b/src/backend/executor/execExprInterp.c @@ -3829,6 +3829,8 @@ ExecEvalXmlExpr(ExprState *state, ExprEvalStep *op) { Datum *argvalue = op->d.xmlexpr.argvalue; bool *argnull = op->d.xmlexpr.argnull; + XmlSerializeFormat format = op->d.xmlexpr.xexpr->format; + text *data; /* argument type is known to be xml */ Assert(list_length(xexpr->args) == 1); @@ -3837,9 +3839,15 @@ ExecEvalXmlExpr(ExprState *state, ExprEvalStep *op) return; value = argvalue[0]; - *op->resvalue = PointerGetDatum(xmltotext_with_xmloption(DatumGetXmlP(value), - xexpr->xmloption)); *op->resnull = false; + + data = xmltotext_with_xmloption(DatumGetXmlP(value), + xexpr->xmloption); + + if (format == XMLDEFAULT_FORMAT) + *op->resvalue = PointerGetDatum(data); + else if (format == XMLCANONICAL || format == XMLCANONICAL_WITH_COMMENTS) + *op->resvalue = PointerGetDatum(xmlserialize_canonical(data,format)); } break; diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index a0138382a1..af5f3dfdfd 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -619,6 +619,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <defelt> xmltable_column_option_el %type <list> xml_namespace_list %type <target> xml_namespace_el +%type <ival> opt_xml_serialize_format %type <node> func_application func_expr_common_subexpr %type <node> func_expr func_expr_windowless @@ -676,7 +677,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); BACKWARD BEFORE BEGIN_P BETWEEN BIGINT BINARY BIT BOOLEAN_P BOTH BREADTH BY - CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P + CACHE CALL CALLED CANONICAL CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT @@ -15532,13 +15533,14 @@ func_expr_common_subexpr: $$ = makeXmlExpr(IS_XMLROOT, NULL, NIL, list_make3($3, $5, $6), @1); } - | XMLSERIALIZE '(' document_or_content a_expr AS SimpleTypename ')' + | XMLSERIALIZE '(' document_or_content a_expr AS SimpleTypename opt_xml_serialize_format ')' { XmlSerialize *n = makeNode(XmlSerialize); n->xmloption = $3; n->expr = $4; n->typeName = $6; + n->format = $7; n->location = @1; $$ = (Node *) n; } @@ -15622,6 +15624,12 @@ xml_passing_mech: | BY VALUE_P ; +opt_xml_serialize_format: + CANONICAL { $$ = XMLCANONICAL; } + | CANONICAL WITH NO COMMENTS { $$ = XMLCANONICAL; } + | CANONICAL WITH COMMENTS { $$ = XMLCANONICAL_WITH_COMMENTS; } + | /*EMPTY*/ { $$ = XMLDEFAULT_FORMAT; } + ; /* * Aggregate decoration clauses @@ -16737,6 +16745,7 @@ unreserved_keyword: | CACHE | CALL | CALLED + | CANONICAL | CASCADE | CASCADED | CATALOG_P @@ -17259,6 +17268,7 @@ bare_label_keyword: | CACHE | CALL | CALLED + | CANONICAL | CASCADE | CASCADED | CASE diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 78221d2e0f..799a345067 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -2332,6 +2332,7 @@ transformXmlSerialize(ParseState *pstate, XmlSerialize *xs) xexpr->xmloption = xs->xmloption; xexpr->location = xs->location; + xexpr->format = xs->format; /* We actually only need these to be able to parse back the expression. */ xexpr->type = targetType; xexpr->typmod = targetTypmod; diff --git a/src/backend/utils/adt/xml.c b/src/backend/utils/adt/xml.c index 079bcb1208..f618be524b 100644 --- a/src/backend/utils/adt/xml.c +++ b/src/backend/utils/adt/xml.c @@ -56,6 +56,7 @@ #include <libxml/xmlwriter.h> #include <libxml/xpath.h> #include <libxml/xpathInternals.h> +#include <libxml/c14n.h> /* * We used to check for xmlStructuredErrorContext via a configure test; but @@ -4818,3 +4819,72 @@ XmlTableDestroyOpaque(TableFuncScanState *state) NO_XML_SUPPORT(); #endif /* not USE_LIBXML */ } + +xmltype * +xmlserialize_canonical(text *data, XmlSerializeFormat format) +{ +#ifdef USE_LIBXML + + xmlDocPtr doc; + xmlChar *xmlbuf = NULL; + xmlChar *encodingStr = NULL; + int nbytes; + int with_comments = 0; /* 0 = remove xml comments (default) */ + int encoding; + StringInfoData buf; + + + if (format != XMLCANONICAL && format != XMLCANONICAL_WITH_COMMENTS) + elog(ERROR,"invalid canonical xml option"); + else if (format == XMLCANONICAL_WITH_COMMENTS) + with_comments = 1; + + /* + * Parses the encoding from the header, so that xml_parse() knows what + * it is dealing with. If not explicitly declared, we assume UTF-8. + */ + parse_xml_decl(xml_text2xmlChar(data), NULL, NULL, &encodingStr, NULL); + encoding = encodingStr ? xmlChar_to_encoding(encodingStr) : PG_UTF8; + + doc = xml_parse(data, XMLOPTION_DOCUMENT, false, encoding, NULL); + + if(!doc) + elog(ERROR, "could not parse the given XML document"); + + /* + * int + * xmlC14NDocDumpMemory ( + * xmlDocPtr doc, # the XML document for canonization + * xmlNodeSetPtr nodes, # the nodes set to be included in the canonized image + * or NULL if all document nodes should be included + * int mode, # 0 = Original C14N 1.0 (Outdated) + * 1 = Exclusive C14N 1.0 (Outdated) + * 2 = C14N 1.1 + * xmlChar **inclusive_ns_prefixes, # the list of inclusive namespace prefixes ended with + * a NULL or NULL if there is no inclusive namespaces + * (only for exclusive canonicalization, ignored otherwise) + * int with_comments, # include comments in the result (!=0) or not (==0) + * xmlChar **xmlbuf # the memory pointer for allocated canonical XML text; + * ) + * Returns: the number of bytes written on success or a negative value on fail. + */ + + nbytes = xmlC14NDocDumpMemory(doc, NULL, 2, NULL, with_comments, &xmlbuf); + + xmlFreeDoc(doc); + + if(nbytes < 0) + elog(ERROR,"could not canonicalize the given XML document"); + + initStringInfo(&buf); + appendStringInfoString(&buf, (const char *) xmlbuf); + + xmlFree(xmlbuf); + + return stringinfo_to_xmltype(&buf); + +#else + NO_XML_SUPPORT(); + return 0; +#endif +} \ No newline at end of file diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index f7d7f10f7d..8ba0984266 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -842,6 +842,7 @@ typedef struct XmlSerialize Node *expr; TypeName *typeName; int location; /* token location, or -1 if unknown */ + XmlSerializeFormat format; /* serialization format */ } XmlSerialize; /* Partitioning related definitions */ diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h index b4292253cc..79fdcc2650 100644 --- a/src/include/nodes/primnodes.h +++ b/src/include/nodes/primnodes.h @@ -1471,6 +1471,13 @@ typedef enum XmlOptionType XMLOPTION_CONTENT } XmlOptionType; +typedef enum XmlSerializeFormat +{ + XMLCANONICAL, /* canonical form without xml comments */ + XMLCANONICAL_WITH_COMMENTS, /* canonical form with xml comments */ + XMLDEFAULT_FORMAT /* unformatted xml representation */ +} XmlSerializeFormat; + typedef struct XmlExpr { Expr xpr; @@ -1491,6 +1498,8 @@ typedef struct XmlExpr int32 typmod pg_node_attr(query_jumble_ignore); /* token location, or -1 if unknown */ int location; + /* serialization format: XMLCANONICAL, XMLCANONICAL_WITH_COMMENTS, XMLINDENT */ + XmlSerializeFormat format pg_node_attr(query_jumble_ignore); } XmlExpr; /* ---------------- diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index bb36213e6f..c1b1a720fe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -67,6 +67,7 @@ PG_KEYWORD("by", BY, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cache", CACHE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("call", CALL, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("called", CALLED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("canonical", CANONICAL, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cascade", CASCADE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cascaded", CASCADED, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("case", CASE, RESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/utils/xml.h b/src/include/utils/xml.h index 311da06cd6..745ebefe24 100644 --- a/src/include/utils/xml.h +++ b/src/include/utils/xml.h @@ -90,4 +90,5 @@ extern PGDLLIMPORT int xmloption; /* XmlOptionType, but int for guc enum */ extern PGDLLIMPORT const TableFuncRoutine XmlTableRoutine; +xmltype *xmlserialize_canonical(text *data, XmlSerializeFormat format); #endif /* XML_H */ diff --git a/src/test/regress/expected/xml.out b/src/test/regress/expected/xml.out index ad852dc2f7..5c205a0d88 100644 --- a/src/test/regress/expected/xml.out +++ b/src/test/regress/expected/xml.out @@ -486,6 +486,114 @@ SELECT xmlserialize(content 'good' as char(10)); SELECT xmlserialize(document 'bad' as text); ERROR: not an XML document +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ?column? ---------- diff --git a/src/test/regress/expected/xml_1.out b/src/test/regress/expected/xml_1.out index 70fe34a04f..cb372c39e2 100644 --- a/src/test/regress/expected/xml_1.out +++ b/src/test/regress/expected/xml_1.out @@ -309,6 +309,110 @@ ERROR: unsupported XML feature LINE 1: SELECT xmlserialize(document 'bad' as text); ^ DETAIL: This functionality requires the server to be built with libxml support. +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +ERROR: unsupported XML feature +LINE 2: (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + ^ +DETAIL: This functionality requires the server to be built with libxml support. +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- +(0 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ERROR: unsupported XML feature LINE 1: SELECT xml '<foo>bar</foo>' IS DOCUMENT; diff --git a/src/test/regress/expected/xml_2.out b/src/test/regress/expected/xml_2.out index 4f029d0072..5a05aad34c 100644 --- a/src/test/regress/expected/xml_2.out +++ b/src/test/regress/expected/xml_2.out @@ -466,6 +466,114 @@ SELECT xmlserialize(content 'good' as char(10)); SELECT xmlserialize(document 'bad' as text); ERROR: not an XML document +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ?column? ---------- diff --git a/src/test/regress/sql/xml.sql b/src/test/regress/sql/xml.sql index 24e40d2653..90ede4b799 100644 --- a/src/test/regress/sql/xml.sql +++ b/src/test/regress/sql/xml.sql @@ -132,6 +132,67 @@ SELECT xmlserialize(content data as character varying(20)) FROM xmltest; SELECT xmlserialize(content 'good' as char(10)); SELECT xmlserialize(document 'bad' as text); +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; SELECT xml '<foo>bar</foo><bar>foo</bar>' IS DOCUMENT; -- 2.25.1 ^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: [PATCH] Add CANONICAL option to xmlserialize 2023-03-05 21:00 Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-05 22:20 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-05 23:32 ` Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-06 10:50 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-06 13:19 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> @ 2023-03-14 07:49 ` Jim Jones <[email protected]> 2023-03-17 09:46 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 0 siblings, 1 reply; 28+ messages in thread From: Jim Jones @ 2023-03-14 07:49 UTC (permalink / raw) To: Thomas Munro <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> v4 attached fixes an encoding issue at the xml_parse call. It now uses GetDatabaseEncoding(). Best, Jim Attachments: [text/x-patch] v4-0001-Add-CANONICAL-format-to-xmlserialize.patch (31.5K, ../../[email protected]/2-v4-0001-Add-CANONICAL-format-to-xmlserialize.patch) download | inline diff: From 3ff8e7bd9a9e43194d834ba6b125841539d5df1c Mon Sep 17 00:00:00 2001 From: Jim Jones <[email protected]> Date: Mon, 6 Mar 2023 14:08:54 +0100 Subject: [PATCH v4] Add CANONICAL format to xmlserialize This patch introduces the CANONICAL option to xmlserialize, which serializes xml documents in their canonical form - as described in the W3C Canonical XML Version 1.1 specification. This option can be used with the additional parameter WITH [NO] COMMENTS to keep or remove xml comments from the canonical xml output. This feature is based on the function xmlC14NDocDumpMemory from the C14N module of libxml2. This patch also includes regression tests and documentation. --- doc/src/sgml/datatype.sgml | 41 +++++++++- src/backend/executor/execExprInterp.c | 13 +++- src/backend/parser/gram.y | 14 +++- src/backend/parser/parse_expr.c | 1 + src/backend/utils/adt/xml.c | 57 ++++++++++++++ src/include/nodes/parsenodes.h | 1 + src/include/nodes/primnodes.h | 9 +++ src/include/parser/kwlist.h | 1 + src/include/utils/xml.h | 1 + src/test/regress/expected/xml.out | 108 ++++++++++++++++++++++++++ src/test/regress/expected/xml_1.out | 104 +++++++++++++++++++++++++ src/test/regress/expected/xml_2.out | 108 ++++++++++++++++++++++++++ src/test/regress/sql/xml.sql | 61 +++++++++++++++ 13 files changed, 514 insertions(+), 5 deletions(-) diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml index 467b49b199..46ec95dbb8 100644 --- a/doc/src/sgml/datatype.sgml +++ b/doc/src/sgml/datatype.sgml @@ -4460,7 +4460,7 @@ xml '<foo>bar</foo>' <type>xml</type>, uses the function <function>xmlserialize</function>:<indexterm><primary>xmlserialize</primary></indexterm> <synopsis> -XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <replaceable>type</replaceable> ) +XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <replaceable>type</replaceable> [ CANONICAL [ WITH [NO] COMMENTS ] ]) </synopsis> <replaceable>type</replaceable> can be <type>character</type>, <type>character varying</type>, or @@ -4470,6 +4470,45 @@ XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <repla you to simply cast the value. </para> + <para> + The option <type>CANONICAL</type> converts a given + XML document to its <ulink url="https://www.w3.org/TR/xml-c14n11/#Terminology">canonical form</ulink> + based on the <ulink url="https://www.w3.org/TR/xml-c14n11/">W3C Canonical XML 1.1 Specification</ulink>. + It is basically designed to provide applications the ability to compare xml documents or test if they + have been changed. The optional parameter <type>WITH [NO] COMMENTS</type> removes or keeps XML comments + from the given document. + </para> + + <para> + Example: + +<screen><![CDATA[ +SELECT + xmlserialize(DOCUMENT + '<foo> + <!-- a comment --> + <bar c="3" b="2" a="1">42</bar> + <empty/> + </foo>'::xml AS text CANONICAL); + xmlserialize +----------------------------------------------------------- + <foo><bar a="1" b="2" c="3">42</bar><empty></empty></foo> +(1 row) + +SELECT + xmlserialize(DOCUMENT + '<foo> + <!-- a comment --> + <bar c="3" b="2" a="1">42</bar> + <empty/> + </foo>'::xml AS text CANONICAL WITH COMMENTS); + xmlserialize +----------------------------------------------------------------------------- + <foo><!-- a comment --><bar a="1" b="2" c="3">42</bar><empty></empty></foo> +(1 row) + +]]></screen> + </para> <para> When a character string value is cast to or from type <type>xml</type> without going through <type>XMLPARSE</type> or diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c index 19351fe34b..ce376a6fcd 100644 --- a/src/backend/executor/execExprInterp.c +++ b/src/backend/executor/execExprInterp.c @@ -3827,18 +3827,27 @@ ExecEvalXmlExpr(ExprState *state, ExprEvalStep *op) case IS_XMLSERIALIZE: { + text *result; Datum *argvalue = op->d.xmlexpr.argvalue; bool *argnull = op->d.xmlexpr.argnull; + XmlSerializeFormat format = op->d.xmlexpr.xexpr->format; /* argument type is known to be xml */ Assert(list_length(xexpr->args) == 1); if (argnull[0]) return; + value = argvalue[0]; - *op->resvalue = PointerGetDatum(xmltotext_with_xmloption(DatumGetXmlP(value), - xexpr->xmloption)); + if (format == XMLCANONICAL || format == XMLCANONICAL_WITH_COMMENTS) + result = xmlserialize_canonical(DatumGetXmlP(value), + format); + else + result = xmltotext_with_xmloption(DatumGetXmlP(value), + xexpr->xmloption); + + *op->resvalue = PointerGetDatum(result); *op->resnull = false; } break; diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index a0138382a1..af5f3dfdfd 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -619,6 +619,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <defelt> xmltable_column_option_el %type <list> xml_namespace_list %type <target> xml_namespace_el +%type <ival> opt_xml_serialize_format %type <node> func_application func_expr_common_subexpr %type <node> func_expr func_expr_windowless @@ -676,7 +677,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); BACKWARD BEFORE BEGIN_P BETWEEN BIGINT BINARY BIT BOOLEAN_P BOTH BREADTH BY - CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P + CACHE CALL CALLED CANONICAL CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT @@ -15532,13 +15533,14 @@ func_expr_common_subexpr: $$ = makeXmlExpr(IS_XMLROOT, NULL, NIL, list_make3($3, $5, $6), @1); } - | XMLSERIALIZE '(' document_or_content a_expr AS SimpleTypename ')' + | XMLSERIALIZE '(' document_or_content a_expr AS SimpleTypename opt_xml_serialize_format ')' { XmlSerialize *n = makeNode(XmlSerialize); n->xmloption = $3; n->expr = $4; n->typeName = $6; + n->format = $7; n->location = @1; $$ = (Node *) n; } @@ -15622,6 +15624,12 @@ xml_passing_mech: | BY VALUE_P ; +opt_xml_serialize_format: + CANONICAL { $$ = XMLCANONICAL; } + | CANONICAL WITH NO COMMENTS { $$ = XMLCANONICAL; } + | CANONICAL WITH COMMENTS { $$ = XMLCANONICAL_WITH_COMMENTS; } + | /*EMPTY*/ { $$ = XMLDEFAULT_FORMAT; } + ; /* * Aggregate decoration clauses @@ -16737,6 +16745,7 @@ unreserved_keyword: | CACHE | CALL | CALLED + | CANONICAL | CASCADE | CASCADED | CATALOG_P @@ -17259,6 +17268,7 @@ bare_label_keyword: | CACHE | CALL | CALLED + | CANONICAL | CASCADE | CASCADED | CASE diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 78221d2e0f..799a345067 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -2332,6 +2332,7 @@ transformXmlSerialize(ParseState *pstate, XmlSerialize *xs) xexpr->xmloption = xs->xmloption; xexpr->location = xs->location; + xexpr->format = xs->format; /* We actually only need these to be able to parse back the expression. */ xexpr->type = targetType; xexpr->typmod = targetTypmod; diff --git a/src/backend/utils/adt/xml.c b/src/backend/utils/adt/xml.c index 079bcb1208..37e2df968a 100644 --- a/src/backend/utils/adt/xml.c +++ b/src/backend/utils/adt/xml.c @@ -56,6 +56,7 @@ #include <libxml/xmlwriter.h> #include <libxml/xpath.h> #include <libxml/xpathInternals.h> +#include <libxml/c14n.h> /* * We used to check for xmlStructuredErrorContext via a configure test; but @@ -4818,3 +4819,59 @@ XmlTableDestroyOpaque(TableFuncScanState *state) NO_XML_SUPPORT(); #endif /* not USE_LIBXML */ } + +text * +xmlserialize_canonical(text *data, XmlSerializeFormat format) +{ +#ifdef USE_LIBXML + + xmlDocPtr doc; + xmlChar *xmlbuf = NULL; + text *result; + int nbytes; + int with_comments = 0; /* 0 = remove xml comments (default) */ + + if (format != XMLCANONICAL && format != XMLCANONICAL_WITH_COMMENTS) + elog(ERROR,"invalid canonical xml option"); + else if (format == XMLCANONICAL_WITH_COMMENTS) + with_comments = 1; + + doc = xml_parse(data, XMLOPTION_DOCUMENT, false, GetDatabaseEncoding(), NULL); + + Assert(doc); + + /* + * xmlC14NDocDumpMemory ( + * xmlDocPtr doc, # the XML document for canonization + * xmlNodeSetPtr nodes, # the nodes set to be included in the canonized image + * or NULL if all document nodes should be included + * int mode, # 0 = Original C14N 1.0 (Outdated) + * 1 = Exclusive C14N 1.0 (Outdated) + * 2 = C14N 1.1 + * xmlChar **inclusive_ns_prefixes, # the list of inclusive namespace prefixes ended with + * a NULL or NULL if there is no inclusive namespaces + * (only for exclusive canonicalization, ignored otherwise) + * int with_comments, # include comments in the result (!=0) or not (==0) + * xmlChar **xmlbuf # the memory pointer for allocated canonical XML text; + * ) + * Returns: the number of bytes written on success or a negative value on fail. + */ + + nbytes = xmlC14NDocDumpMemory(doc, NULL, 2, NULL, with_comments, &xmlbuf); + + xmlFreeDoc(doc); + + if(nbytes < 0) + elog(ERROR,"could not canonicalize the given XML document"); + + result = cstring_to_text_with_len((const char *) xmlbuf, nbytes); + + xmlFree(xmlbuf); + + return result; + +#else + NO_XML_SUPPORT(); + return 0; +#endif +} \ No newline at end of file diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index f7d7f10f7d..8ba0984266 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -842,6 +842,7 @@ typedef struct XmlSerialize Node *expr; TypeName *typeName; int location; /* token location, or -1 if unknown */ + XmlSerializeFormat format; /* serialization format */ } XmlSerialize; /* Partitioning related definitions */ diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h index b4292253cc..79fdcc2650 100644 --- a/src/include/nodes/primnodes.h +++ b/src/include/nodes/primnodes.h @@ -1471,6 +1471,13 @@ typedef enum XmlOptionType XMLOPTION_CONTENT } XmlOptionType; +typedef enum XmlSerializeFormat +{ + XMLCANONICAL, /* canonical form without xml comments */ + XMLCANONICAL_WITH_COMMENTS, /* canonical form with xml comments */ + XMLDEFAULT_FORMAT /* unformatted xml representation */ +} XmlSerializeFormat; + typedef struct XmlExpr { Expr xpr; @@ -1491,6 +1498,8 @@ typedef struct XmlExpr int32 typmod pg_node_attr(query_jumble_ignore); /* token location, or -1 if unknown */ int location; + /* serialization format: XMLCANONICAL, XMLCANONICAL_WITH_COMMENTS, XMLINDENT */ + XmlSerializeFormat format pg_node_attr(query_jumble_ignore); } XmlExpr; /* ---------------- diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index bb36213e6f..c1b1a720fe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -67,6 +67,7 @@ PG_KEYWORD("by", BY, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cache", CACHE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("call", CALL, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("called", CALLED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("canonical", CANONICAL, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cascade", CASCADE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cascaded", CASCADED, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("case", CASE, RESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/utils/xml.h b/src/include/utils/xml.h index 311da06cd6..c894310e18 100644 --- a/src/include/utils/xml.h +++ b/src/include/utils/xml.h @@ -90,4 +90,5 @@ extern PGDLLIMPORT int xmloption; /* XmlOptionType, but int for guc enum */ extern PGDLLIMPORT const TableFuncRoutine XmlTableRoutine; +extern text *xmlserialize_canonical(text *data, XmlSerializeFormat format); #endif /* XML_H */ diff --git a/src/test/regress/expected/xml.out b/src/test/regress/expected/xml.out index ad852dc2f7..c17cc266c7 100644 --- a/src/test/regress/expected/xml.out +++ b/src/test/regress/expected/xml.out @@ -486,6 +486,114 @@ SELECT xmlserialize(content 'good' as char(10)); SELECT xmlserialize(document 'bad' as text); ERROR: not an XML document +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ?column? ---------- diff --git a/src/test/regress/expected/xml_1.out b/src/test/regress/expected/xml_1.out index 70fe34a04f..cb372c39e2 100644 --- a/src/test/regress/expected/xml_1.out +++ b/src/test/regress/expected/xml_1.out @@ -309,6 +309,110 @@ ERROR: unsupported XML feature LINE 1: SELECT xmlserialize(document 'bad' as text); ^ DETAIL: This functionality requires the server to be built with libxml support. +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +ERROR: unsupported XML feature +LINE 2: (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + ^ +DETAIL: This functionality requires the server to be built with libxml support. +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- +(0 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ERROR: unsupported XML feature LINE 1: SELECT xml '<foo>bar</foo>' IS DOCUMENT; diff --git a/src/test/regress/expected/xml_2.out b/src/test/regress/expected/xml_2.out index 4f029d0072..4c51962c18 100644 --- a/src/test/regress/expected/xml_2.out +++ b/src/test/regress/expected/xml_2.out @@ -466,6 +466,114 @@ SELECT xmlserialize(content 'good' as char(10)); SELECT xmlserialize(document 'bad' as text); ERROR: not an XML document +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ?column? ---------- diff --git a/src/test/regress/sql/xml.sql b/src/test/regress/sql/xml.sql index 24e40d2653..90ede4b799 100644 --- a/src/test/regress/sql/xml.sql +++ b/src/test/regress/sql/xml.sql @@ -132,6 +132,67 @@ SELECT xmlserialize(content data as character varying(20)) FROM xmltest; SELECT xmlserialize(content 'good' as char(10)); SELECT xmlserialize(document 'bad' as text); +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; SELECT xml '<foo>bar</foo><bar>foo</bar>' IS DOCUMENT; -- 2.25.1 ^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: [PATCH] Add CANONICAL option to xmlserialize 2023-03-05 21:00 Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-05 22:20 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-05 23:32 ` Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-06 10:50 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-06 13:19 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-14 07:49 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> @ 2023-03-17 09:46 ` Jim Jones <[email protected]> 2023-03-17 12:30 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 0 siblings, 1 reply; 28+ messages in thread From: Jim Jones @ 2023-03-17 09:46 UTC (permalink / raw) To: Thomas Munro <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> v5 attached is a rebase over the latest changes in xmlserialize (INDENT output). Attachments: [text/x-patch] v5-0001-Add-CANONICAL-output-format-to-xmlserialize.patch (41.3K, ../../[email protected]/2-v5-0001-Add-CANONICAL-output-format-to-xmlserialize.patch) download | inline diff: From 24f045ccf7ac000a509910cb32c54ce4c91e2c33 Mon Sep 17 00:00:00 2001 From: Jim Jones <[email protected]> Date: Fri, 17 Mar 2023 10:23:48 +0100 Subject: [PATCH v5] Add CANONICAL output format to xmlserialize This patch introduces the CANONICAL option to xmlserialize, which serializes xml documents in their canonical form - as described in the W3C Canonical XML Version 1.1 specification. This option can be used with the additional parameter WITH [NO] COMMENTS to keep or remove xml comments from the canonical xml output. This feature is based on the function xmlC14NDocDumpMemory from the C14N module of libxml2. This patch also includes regression tests and documentation. --- doc/src/sgml/datatype.sgml | 41 +++- src/backend/executor/execExprInterp.c | 2 +- src/backend/parser/gram.y | 21 ++- src/backend/parser/parse_expr.c | 2 +- src/backend/utils/adt/xml.c | 258 ++++++++++++++++---------- src/include/nodes/parsenodes.h | 1 + src/include/nodes/primnodes.h | 12 +- src/include/parser/kwlist.h | 1 + src/include/utils/xml.h | 2 +- src/test/regress/expected/xml.out | 112 +++++++++++ src/test/regress/expected/xml_1.out | 108 +++++++++++ src/test/regress/expected/xml_2.out | 112 +++++++++++ src/test/regress/sql/xml.sql | 63 +++++++ 13 files changed, 626 insertions(+), 109 deletions(-) diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml index 4df8bd1b64..90415265d4 100644 --- a/doc/src/sgml/datatype.sgml +++ b/doc/src/sgml/datatype.sgml @@ -4460,7 +4460,7 @@ xml '<foo>bar</foo>' <type>xml</type>, uses the function <function>xmlserialize</function>:<indexterm><primary>xmlserialize</primary></indexterm> <synopsis> -XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <replaceable>type</replaceable> [ [ NO ] INDENT ] ) +XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <replaceable>type</replaceable> [ { [ NO ] INDENT ] | CANONICAL [ WITH [NO] COMMENTS ]}) </synopsis> <replaceable>type</replaceable> can be <type>character</type>, <type>character varying</type>, or @@ -4477,6 +4477,45 @@ XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <repla type likewise produces the original string. </para> + <para> + The option <type>CANONICAL</type> converts a given + XML document to its <ulink url="https://www.w3.org/TR/xml-c14n11/#Terminology">canonical form</ulink> + based on the <ulink url="https://www.w3.org/TR/xml-c14n11/">W3C Canonical XML 1.1 Specification</ulink>. + It is basically designed to provide applications the ability to compare xml documents or test if they + have been changed. The optional parameter <type>WITH [NO] COMMENTS</type> removes or keeps XML comments + from the given document. + </para> + + <para> + Example: + +<screen><![CDATA[ +SELECT + xmlserialize(DOCUMENT + '<foo> + <!-- a comment --> + <bar c="3" b="2" a="1">42</bar> + <empty/> + </foo>'::xml AS text CANONICAL); + xmlserialize +----------------------------------------------------------- + <foo><bar a="1" b="2" c="3">42</bar><empty></empty></foo> +(1 row) + +SELECT + xmlserialize(DOCUMENT + '<foo> + <!-- a comment --> + <bar c="3" b="2" a="1">42</bar> + <empty/> + </foo>'::xml AS text CANONICAL WITH COMMENTS); + xmlserialize +----------------------------------------------------------------------------- + <foo><!-- a comment --><bar a="1" b="2" c="3">42</bar><empty></empty></foo> +(1 row) + +]]></screen> + </para> <para> When a character string value is cast to or from type <type>xml</type> without going through <type>XMLPARSE</type> or diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c index 9cb9625ce9..d3dfc9ecfc 100644 --- a/src/backend/executor/execExprInterp.c +++ b/src/backend/executor/execExprInterp.c @@ -3840,7 +3840,7 @@ ExecEvalXmlExpr(ExprState *state, ExprEvalStep *op) *op->resvalue = PointerGetDatum(xmltotext_with_options(DatumGetXmlP(value), xexpr->xmloption, - xexpr->indent)); + xexpr->format)); *op->resnull = false; } break; diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index efe88ccf9d..fd9315feb8 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -613,12 +613,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <node> xml_root_version opt_xml_root_standalone %type <node> xmlexists_argument %type <ival> document_or_content -%type <boolean> xml_indent_option xml_whitespace_option +%type <boolean> xml_whitespace_option %type <list> xmltable_column_list xmltable_column_option_list %type <node> xmltable_column_el %type <defelt> xmltable_column_option_el %type <list> xml_namespace_list %type <target> xml_namespace_el +%type <ival> opt_xml_serialize_format %type <node> func_application func_expr_common_subexpr %type <node> func_expr func_expr_windowless @@ -676,7 +677,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); BACKWARD BEFORE BEGIN_P BETWEEN BIGINT BINARY BIT BOOLEAN_P BOTH BREADTH BY - CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P + CACHE CALL CALLED CANONICAL CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT @@ -15532,14 +15533,14 @@ func_expr_common_subexpr: $$ = makeXmlExpr(IS_XMLROOT, NULL, NIL, list_make3($3, $5, $6), @1); } - | XMLSERIALIZE '(' document_or_content a_expr AS SimpleTypename xml_indent_option ')' + | XMLSERIALIZE '(' document_or_content a_expr AS SimpleTypename opt_xml_serialize_format ')' { XmlSerialize *n = makeNode(XmlSerialize); n->xmloption = $3; n->expr = $4; n->typeName = $6; - n->indent = $7; + n->format = $7; n->location = @1; $$ = (Node *) n; } @@ -15593,9 +15594,13 @@ document_or_content: DOCUMENT_P { $$ = XMLOPTION_DOCUMENT; } | CONTENT_P { $$ = XMLOPTION_CONTENT; } ; -xml_indent_option: INDENT { $$ = true; } - | NO INDENT { $$ = false; } - | /*EMPTY*/ { $$ = false; } +opt_xml_serialize_format: + INDENT { $$ = XMLSERIALIZE_INDENT; } + | NO INDENT { $$ = XMLSERIALIZE_NO_FORMAT; } + | CANONICAL { $$ = XMLSERIALIZE_CANONICAL; } + | CANONICAL WITH NO COMMENTS { $$ = XMLSERIALIZE_CANONICAL; } + | CANONICAL WITH COMMENTS { $$ = XMLSERIALIZE_CANONICAL_WITH_COMMENTS; } + | /*EMPTY*/ { $$ = XMLSERIALIZE_NO_FORMAT; } ; xml_whitespace_option: PRESERVE WHITESPACE_P { $$ = true; } @@ -16743,6 +16748,7 @@ unreserved_keyword: | CACHE | CALL | CALLED + | CANONICAL | CASCADE | CASCADED | CATALOG_P @@ -17266,6 +17272,7 @@ bare_label_keyword: | CACHE | CALL | CALLED + | CANONICAL | CASCADE | CASCADED | CASE diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 2331417552..7e198fc985 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -2331,7 +2331,7 @@ transformXmlSerialize(ParseState *pstate, XmlSerialize *xs) typenameTypeIdAndMod(pstate, xs->typeName, &targetType, &targetTypmod); xexpr->xmloption = xs->xmloption; - xexpr->indent = xs->indent; + xexpr->format = xs->format; xexpr->location = xs->location; /* We actually only need these to be able to parse back the expression. */ xexpr->type = targetType; diff --git a/src/backend/utils/adt/xml.c b/src/backend/utils/adt/xml.c index 15adbd6a01..576b76d4f7 100644 --- a/src/backend/utils/adt/xml.c +++ b/src/backend/utils/adt/xml.c @@ -57,6 +57,7 @@ #include <libxml/xmlwriter.h> #include <libxml/xpath.h> #include <libxml/xpathInternals.h> +#include <libxml/c14n.h> /* * We used to check for xmlStructuredErrorContext via a configure test; but @@ -622,7 +623,7 @@ xmltotext(PG_FUNCTION_ARGS) text * -xmltotext_with_options(xmltype *data, XmlOptionType xmloption_arg, bool indent) +xmltotext_with_options(xmltype *data, XmlOptionType xmloption_arg, XmlSerializeFormat format) { #ifdef USE_LIBXML text *volatile result; @@ -635,7 +636,7 @@ xmltotext_with_options(xmltype *data, XmlOptionType xmloption_arg, bool indent) PgXmlErrorContext *xmlerrcxt; #endif - if (xmloption_arg != XMLOPTION_DOCUMENT && !indent) + if (xmloption_arg != XMLOPTION_DOCUMENT && format == XMLSERIALIZE_NO_FORMAT) { /* * We don't actually need to do anything, so just return the @@ -661,7 +662,7 @@ xmltotext_with_options(xmltype *data, XmlOptionType xmloption_arg, bool indent) } /* If we weren't asked to indent, we're done. */ - if (!indent) + if (format == XMLSERIALIZE_NO_FORMAT) { xmlFreeDoc(doc); return (text *) data; @@ -670,130 +671,195 @@ xmltotext_with_options(xmltype *data, XmlOptionType xmloption_arg, bool indent) /* Otherwise, we gotta spin up some error handling. */ xmlerrcxt = pg_xml_init(PG_XML_STRICTNESS_ALL); - PG_TRY(); + if(format == XMLSERIALIZE_INDENT) { - size_t decl_len = 0; - - /* The serialized data will go into this buffer. */ - buf = xmlBufferCreate(); - - if (buf == NULL || xmlerrcxt->err_occurred) - xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, - "could not allocate xmlBuffer"); - - /* Detect whether there's an XML declaration */ - parse_xml_decl(xml_text2xmlChar(data), &decl_len, NULL, NULL, NULL); - - /* - * Emit declaration only if the input had one. Note: some versions of - * xmlSaveToBuffer leak memory if a non-null encoding argument is - * passed, so don't do that. We don't want any encoding conversion - * anyway. - */ - if (decl_len == 0) - ctxt = xmlSaveToBuffer(buf, NULL, - XML_SAVE_NO_DECL | XML_SAVE_FORMAT); - else - ctxt = xmlSaveToBuffer(buf, NULL, - XML_SAVE_FORMAT); - - if (ctxt == NULL || xmlerrcxt->err_occurred) - xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, - "could not allocate xmlSaveCtxt"); - - if (parsed_xmloptiontype == XMLOPTION_DOCUMENT) - { - /* If it's a document, saving is easy. */ - if (xmlSaveDoc(ctxt, doc) == -1 || xmlerrcxt->err_occurred) - xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, - "could not save document to xmlBuffer"); - } - else if (content_nodes != NULL) + PG_TRY(); { - /* - * Deal with the case where we have non-singly-rooted XML. - * libxml's dump functions don't work well for that without help. - * We build a fake root node that serves as a container for the - * content nodes, and then iterate over the nodes. - */ - xmlNodePtr root; - xmlNodePtr newline; + size_t decl_len = 0; + + /* The serialized data will go into this buffer. */ + buf = xmlBufferCreate(); - root = xmlNewNode(NULL, (const xmlChar *) "content-root"); - if (root == NULL || xmlerrcxt->err_occurred) + if (buf == NULL || xmlerrcxt->err_occurred) xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, - "could not allocate xml node"); + "could not allocate xmlBuffer"); - /* This attaches root to doc, so we need not free it separately. */ - xmlDocSetRootElement(doc, root); - xmlAddChild(root, content_nodes); + /* Detect whether there's an XML declaration */ + parse_xml_decl(xml_text2xmlChar(data), &decl_len, NULL, NULL, NULL); /* - * We use this node to insert newlines in the dump. Note: in at - * least some libxml versions, xmlNewDocText would not attach the - * node to the document even if we passed it. Therefore, manage - * freeing of this node manually, and pass NULL here to make sure - * there's not a dangling link. - */ - newline = xmlNewDocText(NULL, (const xmlChar *) "\n"); - if (newline == NULL || xmlerrcxt->err_occurred) + * Emit declaration only if the input had one. Note: some versions of + * xmlSaveToBuffer leak memory if a non-null encoding argument is + * passed, so don't do that. We don't want any encoding conversion + * anyway. + */ + if (decl_len == 0) + ctxt = xmlSaveToBuffer(buf, NULL, + XML_SAVE_NO_DECL | XML_SAVE_FORMAT); + else + ctxt = xmlSaveToBuffer(buf, NULL, + XML_SAVE_FORMAT); + + if (ctxt == NULL || xmlerrcxt->err_occurred) xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, - "could not allocate xml node"); + "could not allocate xmlSaveCtxt"); - for (xmlNodePtr node = root->children; node; node = node->next) + if (parsed_xmloptiontype == XMLOPTION_DOCUMENT) + { + /* If it's a document, saving is easy. */ + if (xmlSaveDoc(ctxt, doc) == -1 || xmlerrcxt->err_occurred) + xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, + "could not save document to xmlBuffer"); + } + else if (content_nodes != NULL) { - /* insert newlines between nodes */ - if (node->type != XML_TEXT_NODE && node->prev != NULL) + /* + * Deal with the case where we have non-singly-rooted XML. + * libxml's dump functions don't work well for that without help. + * We build a fake root node that serves as a container for the + * content nodes, and then iterate over the nodes. + */ + xmlNodePtr root; + xmlNodePtr newline; + + root = xmlNewNode(NULL, (const xmlChar *) "content-root"); + if (root == NULL || xmlerrcxt->err_occurred) + xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, + "could not allocate xml node"); + + /* This attaches root to doc, so we need not free it separately. */ + xmlDocSetRootElement(doc, root); + xmlAddChild(root, content_nodes); + + /* + * We use this node to insert newlines in the dump. Note: in at + * least some libxml versions, xmlNewDocText would not attach the + * node to the document even if we passed it. Therefore, manage + * freeing of this node manually, and pass NULL here to make sure + * there's not a dangling link. + */ + newline = xmlNewDocText(NULL, (const xmlChar *) "\n"); + if (newline == NULL || xmlerrcxt->err_occurred) + xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, + "could not allocate xml node"); + + for (xmlNodePtr node = root->children; node; node = node->next) { - if (xmlSaveTree(ctxt, newline) == -1 || xmlerrcxt->err_occurred) + /* insert newlines between nodes */ + if (node->type != XML_TEXT_NODE && node->prev != NULL) + { + if (xmlSaveTree(ctxt, newline) == -1 || xmlerrcxt->err_occurred) + { + xmlFreeNode(newline); + xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, + "could not save newline to xmlBuffer"); + } + } + + if (xmlSaveTree(ctxt, node) == -1 || xmlerrcxt->err_occurred) { xmlFreeNode(newline); xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, - "could not save newline to xmlBuffer"); + "could not save content to xmlBuffer"); } } - if (xmlSaveTree(ctxt, node) == -1 || xmlerrcxt->err_occurred) - { - xmlFreeNode(newline); - xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, - "could not save content to xmlBuffer"); - } + xmlFreeNode(newline); } - xmlFreeNode(newline); - } + if (xmlSaveClose(ctxt) == -1 || xmlerrcxt->err_occurred) + { + ctxt = NULL; /* don't try to close it again */ + xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, + "could not close xmlSaveCtxtPtr"); + } - if (xmlSaveClose(ctxt) == -1 || xmlerrcxt->err_occurred) + result = (text *) xmlBuffer_to_xmltype(buf); + } + PG_CATCH(); { - ctxt = NULL; /* don't try to close it again */ - xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, - "could not close xmlSaveCtxtPtr"); + if (ctxt) + xmlSaveClose(ctxt); + if (buf) + xmlBufferFree(buf); + if (doc) + xmlFreeDoc(doc); + + pg_xml_done(xmlerrcxt, true); + + PG_RE_THROW(); } + PG_END_TRY(); + + xmlBufferFree(buf); + xmlFreeDoc(doc); - result = (text *) xmlBuffer_to_xmltype(buf); + pg_xml_done(xmlerrcxt, false); } - PG_CATCH(); + else if (format == XMLSERIALIZE_CANONICAL || format == XMLSERIALIZE_CANONICAL_WITH_COMMENTS) { - if (ctxt) - xmlSaveClose(ctxt); - if (buf) - xmlBufferFree(buf); - if (doc) - xmlFreeDoc(doc); + xmlChar *xmlbuf = NULL; + int nbytes; + int with_comments = 0; /* 0 = no xml comments (default) */ - pg_xml_done(xmlerrcxt, true); + PG_TRY(); + { + /* 1 = keeps xml comments */ + if (format == XMLSERIALIZE_CANONICAL_WITH_COMMENTS) + with_comments = 1; - PG_RE_THROW(); - } - PG_END_TRY(); + /* Only singly-rooted XML are supported */ + doc = xml_parse(data, XMLOPTION_DOCUMENT, false, + GetDatabaseEncoding(), NULL, NULL, NULL); - xmlBufferFree(buf); - xmlFreeDoc(doc); + if (doc == NULL || escontext.error_occurred) + { + if (doc) + xmlFreeDoc(doc); + /* A soft error must be failure to conform to XMLOPTION_DOCUMENT */ + ereport(ERROR, + (errcode(ERRCODE_NOT_AN_XML_DOCUMENT), + errmsg("not an XML document"))); + } - pg_xml_done(xmlerrcxt, false); + /* + * This dumps the canonicalized XML doc into the xmlChar* buffer. + * mode = 2 means the doc will be canonicalized using the C14N 1.1 standard. + */ + nbytes = xmlC14NDocDumpMemory(doc, NULL, 2, NULL, with_comments, &xmlbuf); + + if(nbytes < 0 || escontext.error_occurred) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + errmsg("could not canonicalize the given XML document"))); + + result = cstring_to_text_with_len((const char *) xmlbuf, nbytes); + } + PG_CATCH(); + { + if (ctxt) + xmlSaveClose(ctxt); + if (xmlbuf) + xmlFree(xmlbuf); + if (doc) + xmlFreeDoc(doc); + + pg_xml_done(xmlerrcxt, true); + + PG_RE_THROW(); + } + PG_END_TRY(); + + xmlFreeDoc(doc); + xmlFree(xmlbuf); + + pg_xml_done(xmlerrcxt, false); + } + else + elog(ERROR,"invalid xmlserialize option"); return result; + #else NO_XML_SUPPORT(); return NULL; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 028588fb33..919d7158f4 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -841,6 +841,7 @@ typedef struct XmlSerialize Node *expr; TypeName *typeName; bool indent; /* [NO] INDENT */ + XmlSerializeFormat format; /* serialization format */ int location; /* token location, or -1 if unknown */ } XmlSerialize; diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h index 8fb5b4b919..2cdb1d970e 100644 --- a/src/include/nodes/primnodes.h +++ b/src/include/nodes/primnodes.h @@ -1474,6 +1474,14 @@ typedef enum XmlOptionType XMLOPTION_CONTENT } XmlOptionType; +typedef enum XmlSerializeFormat +{ + XMLSERIALIZE_INDENT, /* pretty-printed xml serialization */ + XMLSERIALIZE_CANONICAL, /* canonical form without xml comments */ + XMLSERIALIZE_CANONICAL_WITH_COMMENTS, /* canonical form with xml comments */ + XMLSERIALIZE_NO_FORMAT /* unformatted xml representation */ +} XmlSerializeFormat; + typedef struct XmlExpr { Expr xpr; @@ -1489,13 +1497,13 @@ typedef struct XmlExpr List *args; /* DOCUMENT or CONTENT */ XmlOptionType xmloption pg_node_attr(query_jumble_ignore); - /* INDENT option for XMLSERIALIZE */ - bool indent; /* target type/typmod for XMLSERIALIZE */ Oid type pg_node_attr(query_jumble_ignore); int32 typmod pg_node_attr(query_jumble_ignore); /* token location, or -1 if unknown */ int location; + /* serialization format: XMLCANONICAL, XMLCANONICAL_WITH_COMMENTS, XMLINDENT */ + XmlSerializeFormat format pg_node_attr(query_jumble_ignore); } XmlExpr; /* ---------------- diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 753e9ee174..8280f605d1 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -67,6 +67,7 @@ PG_KEYWORD("by", BY, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cache", CACHE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("call", CALL, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("called", CALLED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("canonical", CANONICAL, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cascade", CASCADE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cascaded", CASCADED, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("case", CASE, RESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/utils/xml.h b/src/include/utils/xml.h index 224f6d75ff..b74f216148 100644 --- a/src/include/utils/xml.h +++ b/src/include/utils/xml.h @@ -78,7 +78,7 @@ extern xmltype *xmlpi(const char *target, text *arg, bool arg_is_null, bool *res extern xmltype *xmlroot(xmltype *data, text *version, int standalone); extern bool xml_is_document(xmltype *arg); extern text *xmltotext_with_options(xmltype *data, XmlOptionType xmloption_arg, - bool indent); + XmlSerializeFormat format); extern char *escape_xml(const char *str); extern char *map_sql_identifier_to_xml_name(const char *ident, bool fully_escaped, bool escape_period); diff --git a/src/test/regress/expected/xml.out b/src/test/regress/expected/xml.out index 398345ca67..a73ce5a0cf 100644 --- a/src/test/regress/expected/xml.out +++ b/src/test/regress/expected/xml.out @@ -672,6 +672,118 @@ SELECT xmlserialize(CONTENT '<foo><bar><val x="y">42</val></bar></foo>' AS text t (1 row) +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(DOCUMENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +ERROR: syntax error at or near "INDENT" at character 75 +SELECT xmlserialize(CONTENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +ERROR: syntax error at or near "INDENT" at character 74 +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ?column? ---------- diff --git a/src/test/regress/expected/xml_1.out b/src/test/regress/expected/xml_1.out index 63b779470f..481badaa84 100644 --- a/src/test/regress/expected/xml_1.out +++ b/src/test/regress/expected/xml_1.out @@ -443,6 +443,114 @@ ERROR: unsupported XML feature LINE 1: SELECT xmlserialize(CONTENT '<foo><bar><val x="y">42</val><... ^ DETAIL: This functionality requires the server to be built with libxml support. +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +ERROR: unsupported XML feature +LINE 2: (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + ^ +DETAIL: This functionality requires the server to be built with libxml support. +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- +(0 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +SELECT xmlserialize(DOCUMENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +ERROR: syntax error at or near "INDENT" at character 75 +SELECT xmlserialize(CONTENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +ERROR: syntax error at or near "INDENT" at character 74 +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ERROR: unsupported XML feature LINE 1: SELECT xml '<foo>bar</foo>' IS DOCUMENT; diff --git a/src/test/regress/expected/xml_2.out b/src/test/regress/expected/xml_2.out index 43c2558352..384c3ef946 100644 --- a/src/test/regress/expected/xml_2.out +++ b/src/test/regress/expected/xml_2.out @@ -652,6 +652,118 @@ SELECT xmlserialize(CONTENT '<foo><bar><val x="y">42</val></bar></foo>' AS text t (1 row) +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: not an XML document +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(DOCUMENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +ERROR: syntax error at or near "INDENT" at character 75 +SELECT xmlserialize(CONTENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +ERROR: syntax error at or near "INDENT" at character 74 +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ?column? ---------- diff --git a/src/test/regress/sql/xml.sql b/src/test/regress/sql/xml.sql index a591eea2e5..a3eda9e84f 100644 --- a/src/test/regress/sql/xml.sql +++ b/src/test/regress/sql/xml.sql @@ -168,6 +168,69 @@ SELECT xmlserialize(CONTENT '<foo><bar></bar></foo>' AS text INDENT); -- 'no indent' = not using 'no indent' SELECT xmlserialize(DOCUMENT '<foo><bar><val x="y">42</val></bar></foo>' AS text) = xmlserialize(DOCUMENT '<foo><bar><val x="y">42</val></bar></foo>' AS text NO INDENT); SELECT xmlserialize(CONTENT '<foo><bar><val x="y">42</val></bar></foo>' AS text) = xmlserialize(CONTENT '<foo><bar><val x="y">42</val></bar></foo>' AS text NO INDENT); +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +SELECT xmlserialize(DOCUMENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +SELECT xmlserialize(CONTENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; SELECT xml '<foo>bar</foo><bar>foo</bar>' IS DOCUMENT; -- 2.25.1 ^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: [PATCH] Add CANONICAL option to xmlserialize 2023-03-05 21:00 Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-05 22:20 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-05 23:32 ` Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-06 10:50 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-06 13:19 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-14 07:49 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-17 09:46 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> @ 2023-03-17 12:30 ` Jim Jones <[email protected]> 2023-09-14 11:54 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 0 siblings, 1 reply; 28+ messages in thread From: Jim Jones @ 2023-03-17 12:30 UTC (permalink / raw) To: Thomas Munro <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> After some more testing I realized that v5 was leaking the xmlDocPtr. Now fixed in v6. Attachments: [text/x-patch] v6-0001-Add-CANONICAL-output-format-to-xmlserialize.patch (42.3K, ../../[email protected]/2-v6-0001-Add-CANONICAL-output-format-to-xmlserialize.patch) download | inline diff: From d04d8fdcbedbd5ed88469bd22e079467c26ab7a4 Mon Sep 17 00:00:00 2001 From: Jim Jones <[email protected]> Date: Fri, 17 Mar 2023 10:23:48 +0100 Subject: [PATCH v6] Add CANONICAL output format to xmlserialize This patch introduces the CANONICAL option to xmlserialize, which serializes xml documents in their canonical form - as described in the W3C Canonical XML Version 1.1 specification. This option can be used with the additional parameter WITH [NO] COMMENTS to keep or remove xml comments from the canonical xml output. This feature is based on the function xmlC14NDocDumpMemory from the C14N module of libxml2. This patch also includes regression tests and documentation. --- doc/src/sgml/datatype.sgml | 41 +++- src/backend/executor/execExprInterp.c | 2 +- src/backend/parser/gram.y | 21 +- src/backend/parser/parse_expr.c | 2 +- src/backend/utils/adt/xml.c | 275 ++++++++++++++++---------- src/include/nodes/parsenodes.h | 1 + src/include/nodes/primnodes.h | 12 +- src/include/parser/kwlist.h | 1 + src/include/utils/xml.h | 2 +- src/test/regress/expected/xml.out | 112 +++++++++++ src/test/regress/expected/xml_1.out | 108 ++++++++++ src/test/regress/expected/xml_2.out | 112 +++++++++++ src/test/regress/sql/xml.sql | 63 ++++++ 13 files changed, 639 insertions(+), 113 deletions(-) diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml index 4df8bd1b64..90415265d4 100644 --- a/doc/src/sgml/datatype.sgml +++ b/doc/src/sgml/datatype.sgml @@ -4460,7 +4460,7 @@ xml '<foo>bar</foo>' <type>xml</type>, uses the function <function>xmlserialize</function>:<indexterm><primary>xmlserialize</primary></indexterm> <synopsis> -XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <replaceable>type</replaceable> [ [ NO ] INDENT ] ) +XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <replaceable>type</replaceable> [ { [ NO ] INDENT ] | CANONICAL [ WITH [NO] COMMENTS ]}) </synopsis> <replaceable>type</replaceable> can be <type>character</type>, <type>character varying</type>, or @@ -4477,6 +4477,45 @@ XMLSERIALIZE ( { DOCUMENT | CONTENT } <replaceable>value</replaceable> AS <repla type likewise produces the original string. </para> + <para> + The option <type>CANONICAL</type> converts a given + XML document to its <ulink url="https://www.w3.org/TR/xml-c14n11/#Terminology">canonical form</ulink> + based on the <ulink url="https://www.w3.org/TR/xml-c14n11/">W3C Canonical XML 1.1 Specification</ulink>. + It is basically designed to provide applications the ability to compare xml documents or test if they + have been changed. The optional parameter <type>WITH [NO] COMMENTS</type> removes or keeps XML comments + from the given document. + </para> + + <para> + Example: + +<screen><![CDATA[ +SELECT + xmlserialize(DOCUMENT + '<foo> + <!-- a comment --> + <bar c="3" b="2" a="1">42</bar> + <empty/> + </foo>'::xml AS text CANONICAL); + xmlserialize +----------------------------------------------------------- + <foo><bar a="1" b="2" c="3">42</bar><empty></empty></foo> +(1 row) + +SELECT + xmlserialize(DOCUMENT + '<foo> + <!-- a comment --> + <bar c="3" b="2" a="1">42</bar> + <empty/> + </foo>'::xml AS text CANONICAL WITH COMMENTS); + xmlserialize +----------------------------------------------------------------------------- + <foo><!-- a comment --><bar a="1" b="2" c="3">42</bar><empty></empty></foo> +(1 row) + +]]></screen> + </para> <para> When a character string value is cast to or from type <type>xml</type> without going through <type>XMLPARSE</type> or diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c index 9cb9625ce9..d3dfc9ecfc 100644 --- a/src/backend/executor/execExprInterp.c +++ b/src/backend/executor/execExprInterp.c @@ -3840,7 +3840,7 @@ ExecEvalXmlExpr(ExprState *state, ExprEvalStep *op) *op->resvalue = PointerGetDatum(xmltotext_with_options(DatumGetXmlP(value), xexpr->xmloption, - xexpr->indent)); + xexpr->format)); *op->resnull = false; } break; diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index efe88ccf9d..fd9315feb8 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -613,12 +613,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type <node> xml_root_version opt_xml_root_standalone %type <node> xmlexists_argument %type <ival> document_or_content -%type <boolean> xml_indent_option xml_whitespace_option +%type <boolean> xml_whitespace_option %type <list> xmltable_column_list xmltable_column_option_list %type <node> xmltable_column_el %type <defelt> xmltable_column_option_el %type <list> xml_namespace_list %type <target> xml_namespace_el +%type <ival> opt_xml_serialize_format %type <node> func_application func_expr_common_subexpr %type <node> func_expr func_expr_windowless @@ -676,7 +677,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); BACKWARD BEFORE BEGIN_P BETWEEN BIGINT BINARY BIT BOOLEAN_P BOTH BREADTH BY - CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P + CACHE CALL CALLED CANONICAL CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT @@ -15532,14 +15533,14 @@ func_expr_common_subexpr: $$ = makeXmlExpr(IS_XMLROOT, NULL, NIL, list_make3($3, $5, $6), @1); } - | XMLSERIALIZE '(' document_or_content a_expr AS SimpleTypename xml_indent_option ')' + | XMLSERIALIZE '(' document_or_content a_expr AS SimpleTypename opt_xml_serialize_format ')' { XmlSerialize *n = makeNode(XmlSerialize); n->xmloption = $3; n->expr = $4; n->typeName = $6; - n->indent = $7; + n->format = $7; n->location = @1; $$ = (Node *) n; } @@ -15593,9 +15594,13 @@ document_or_content: DOCUMENT_P { $$ = XMLOPTION_DOCUMENT; } | CONTENT_P { $$ = XMLOPTION_CONTENT; } ; -xml_indent_option: INDENT { $$ = true; } - | NO INDENT { $$ = false; } - | /*EMPTY*/ { $$ = false; } +opt_xml_serialize_format: + INDENT { $$ = XMLSERIALIZE_INDENT; } + | NO INDENT { $$ = XMLSERIALIZE_NO_FORMAT; } + | CANONICAL { $$ = XMLSERIALIZE_CANONICAL; } + | CANONICAL WITH NO COMMENTS { $$ = XMLSERIALIZE_CANONICAL; } + | CANONICAL WITH COMMENTS { $$ = XMLSERIALIZE_CANONICAL_WITH_COMMENTS; } + | /*EMPTY*/ { $$ = XMLSERIALIZE_NO_FORMAT; } ; xml_whitespace_option: PRESERVE WHITESPACE_P { $$ = true; } @@ -16743,6 +16748,7 @@ unreserved_keyword: | CACHE | CALL | CALLED + | CANONICAL | CASCADE | CASCADED | CATALOG_P @@ -17266,6 +17272,7 @@ bare_label_keyword: | CACHE | CALL | CALLED + | CANONICAL | CASCADE | CASCADED | CASE diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c index 2331417552..7e198fc985 100644 --- a/src/backend/parser/parse_expr.c +++ b/src/backend/parser/parse_expr.c @@ -2331,7 +2331,7 @@ transformXmlSerialize(ParseState *pstate, XmlSerialize *xs) typenameTypeIdAndMod(pstate, xs->typeName, &targetType, &targetTypmod); xexpr->xmloption = xs->xmloption; - xexpr->indent = xs->indent; + xexpr->format = xs->format; xexpr->location = xs->location; /* We actually only need these to be able to parse back the expression. */ xexpr->type = targetType; diff --git a/src/backend/utils/adt/xml.c b/src/backend/utils/adt/xml.c index 15adbd6a01..fbce16464a 100644 --- a/src/backend/utils/adt/xml.c +++ b/src/backend/utils/adt/xml.c @@ -57,6 +57,7 @@ #include <libxml/xmlwriter.h> #include <libxml/xpath.h> #include <libxml/xpathInternals.h> +#include <libxml/c14n.h> /* * We used to check for xmlStructuredErrorContext via a configure test; but @@ -622,7 +623,7 @@ xmltotext(PG_FUNCTION_ARGS) text * -xmltotext_with_options(xmltype *data, XmlOptionType xmloption_arg, bool indent) +xmltotext_with_options(xmltype *data, XmlOptionType xmloption_arg, XmlSerializeFormat format) { #ifdef USE_LIBXML text *volatile result; @@ -635,7 +636,7 @@ xmltotext_with_options(xmltype *data, XmlOptionType xmloption_arg, bool indent) PgXmlErrorContext *xmlerrcxt; #endif - if (xmloption_arg != XMLOPTION_DOCUMENT && !indent) + if (xmloption_arg != XMLOPTION_DOCUMENT && format == XMLSERIALIZE_NO_FORMAT) { /* * We don't actually need to do anything, so just return the @@ -646,10 +647,23 @@ xmltotext_with_options(xmltype *data, XmlOptionType xmloption_arg, bool indent) } #ifdef USE_LIBXML - /* Parse the input according to the xmloption */ - doc = xml_parse(data, xmloption_arg, true, GetDatabaseEncoding(), - &parsed_xmloptiontype, &content_nodes, - (Node *) &escontext); + /* + * Parse the input according to the xmloption. + * XML canonical expects a well-formed XML input, so here in case of + * XMLSERIALIZE_CANONICAL or XMLSERIALIZE_CANONICAL_WITH_COMMENTS we + * force xml_parse() to parse 'data' as XMLOPTION_DOCUMENT despite + * of the XmlOptionType given in 'xmloption_arg'. This enables the + * canonicalization of CONTENT fragments if they contain a singly-rooted + * XML - xml_parse() will thrown an error otherwise. + */ + if(format == XMLSERIALIZE_CANONICAL || format == XMLSERIALIZE_CANONICAL_WITH_COMMENTS) + doc = xml_parse(data, XMLOPTION_DOCUMENT, false, + GetDatabaseEncoding(), NULL, NULL, NULL); + else + doc = xml_parse(data, xmloption_arg, true, GetDatabaseEncoding(), + &parsed_xmloptiontype, &content_nodes, + (Node *) &escontext); + if (doc == NULL || escontext.error_occurred) { if (doc) @@ -661,7 +675,7 @@ xmltotext_with_options(xmltype *data, XmlOptionType xmloption_arg, bool indent) } /* If we weren't asked to indent, we're done. */ - if (!indent) + if (format == XMLSERIALIZE_NO_FORMAT) { xmlFreeDoc(doc); return (text *) data; @@ -670,130 +684,191 @@ xmltotext_with_options(xmltype *data, XmlOptionType xmloption_arg, bool indent) /* Otherwise, we gotta spin up some error handling. */ xmlerrcxt = pg_xml_init(PG_XML_STRICTNESS_ALL); - PG_TRY(); + if(format == XMLSERIALIZE_INDENT) { - size_t decl_len = 0; - - /* The serialized data will go into this buffer. */ - buf = xmlBufferCreate(); - - if (buf == NULL || xmlerrcxt->err_occurred) - xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, - "could not allocate xmlBuffer"); - - /* Detect whether there's an XML declaration */ - parse_xml_decl(xml_text2xmlChar(data), &decl_len, NULL, NULL, NULL); - - /* - * Emit declaration only if the input had one. Note: some versions of - * xmlSaveToBuffer leak memory if a non-null encoding argument is - * passed, so don't do that. We don't want any encoding conversion - * anyway. - */ - if (decl_len == 0) - ctxt = xmlSaveToBuffer(buf, NULL, - XML_SAVE_NO_DECL | XML_SAVE_FORMAT); - else - ctxt = xmlSaveToBuffer(buf, NULL, - XML_SAVE_FORMAT); - - if (ctxt == NULL || xmlerrcxt->err_occurred) - xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, - "could not allocate xmlSaveCtxt"); - - if (parsed_xmloptiontype == XMLOPTION_DOCUMENT) - { - /* If it's a document, saving is easy. */ - if (xmlSaveDoc(ctxt, doc) == -1 || xmlerrcxt->err_occurred) - xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, - "could not save document to xmlBuffer"); - } - else if (content_nodes != NULL) + PG_TRY(); { - /* - * Deal with the case where we have non-singly-rooted XML. - * libxml's dump functions don't work well for that without help. - * We build a fake root node that serves as a container for the - * content nodes, and then iterate over the nodes. - */ - xmlNodePtr root; - xmlNodePtr newline; + size_t decl_len = 0; - root = xmlNewNode(NULL, (const xmlChar *) "content-root"); - if (root == NULL || xmlerrcxt->err_occurred) + /* The serialized data will go into this buffer. */ + buf = xmlBufferCreate(); + + if (buf == NULL || xmlerrcxt->err_occurred) xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, - "could not allocate xml node"); + "could not allocate xmlBuffer"); - /* This attaches root to doc, so we need not free it separately. */ - xmlDocSetRootElement(doc, root); - xmlAddChild(root, content_nodes); + /* Detect whether there's an XML declaration */ + parse_xml_decl(xml_text2xmlChar(data), &decl_len, NULL, NULL, NULL); /* - * We use this node to insert newlines in the dump. Note: in at - * least some libxml versions, xmlNewDocText would not attach the - * node to the document even if we passed it. Therefore, manage - * freeing of this node manually, and pass NULL here to make sure - * there's not a dangling link. - */ - newline = xmlNewDocText(NULL, (const xmlChar *) "\n"); - if (newline == NULL || xmlerrcxt->err_occurred) + * Emit declaration only if the input had one. Note: some versions of + * xmlSaveToBuffer leak memory if a non-null encoding argument is + * passed, so don't do that. We don't want any encoding conversion + * anyway. + */ + if (decl_len == 0) + ctxt = xmlSaveToBuffer(buf, NULL, + XML_SAVE_NO_DECL | XML_SAVE_FORMAT); + else + ctxt = xmlSaveToBuffer(buf, NULL, + XML_SAVE_FORMAT); + + if (ctxt == NULL || xmlerrcxt->err_occurred) xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, - "could not allocate xml node"); + "could not allocate xmlSaveCtxt"); - for (xmlNodePtr node = root->children; node; node = node->next) + if (parsed_xmloptiontype == XMLOPTION_DOCUMENT) { - /* insert newlines between nodes */ - if (node->type != XML_TEXT_NODE && node->prev != NULL) + /* If it's a document, saving is easy. */ + if (xmlSaveDoc(ctxt, doc) == -1 || xmlerrcxt->err_occurred) + xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, + "could not save document to xmlBuffer"); + } + else if (content_nodes != NULL) + { + /* + * Deal with the case where we have non-singly-rooted XML. + * libxml's dump functions don't work well for that without help. + * We build a fake root node that serves as a container for the + * content nodes, and then iterate over the nodes. + */ + xmlNodePtr root; + xmlNodePtr newline; + + root = xmlNewNode(NULL, (const xmlChar *) "content-root"); + if (root == NULL || xmlerrcxt->err_occurred) + xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, + "could not allocate xml node"); + + /* This attaches root to doc, so we need not free it separately. */ + xmlDocSetRootElement(doc, root); + xmlAddChild(root, content_nodes); + + /* + * We use this node to insert newlines in the dump. Note: in at + * least some libxml versions, xmlNewDocText would not attach the + * node to the document even if we passed it. Therefore, manage + * freeing of this node manually, and pass NULL here to make sure + * there's not a dangling link. + */ + newline = xmlNewDocText(NULL, (const xmlChar *) "\n"); + if (newline == NULL || xmlerrcxt->err_occurred) + xml_ereport(xmlerrcxt, ERROR, ERRCODE_OUT_OF_MEMORY, + "could not allocate xml node"); + + for (xmlNodePtr node = root->children; node; node = node->next) { - if (xmlSaveTree(ctxt, newline) == -1 || xmlerrcxt->err_occurred) + /* insert newlines between nodes */ + if (node->type != XML_TEXT_NODE && node->prev != NULL) + { + if (xmlSaveTree(ctxt, newline) == -1 || xmlerrcxt->err_occurred) + { + xmlFreeNode(newline); + xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, + "could not save newline to xmlBuffer"); + } + } + + if (xmlSaveTree(ctxt, node) == -1 || xmlerrcxt->err_occurred) { xmlFreeNode(newline); xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, - "could not save newline to xmlBuffer"); + "could not save content to xmlBuffer"); } } - if (xmlSaveTree(ctxt, node) == -1 || xmlerrcxt->err_occurred) - { - xmlFreeNode(newline); - xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, - "could not save content to xmlBuffer"); - } + xmlFreeNode(newline); } - xmlFreeNode(newline); - } + if (xmlSaveClose(ctxt) == -1 || xmlerrcxt->err_occurred) + { + ctxt = NULL; /* don't try to close it again */ + xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, + "could not close xmlSaveCtxtPtr"); + } - if (xmlSaveClose(ctxt) == -1 || xmlerrcxt->err_occurred) + result = (text *) xmlBuffer_to_xmltype(buf); + } + PG_CATCH(); { - ctxt = NULL; /* don't try to close it again */ - xml_ereport(xmlerrcxt, ERROR, ERRCODE_INTERNAL_ERROR, - "could not close xmlSaveCtxtPtr"); + if (ctxt) + xmlSaveClose(ctxt); + if (buf) + xmlBufferFree(buf); + if (doc) + xmlFreeDoc(doc); + + pg_xml_done(xmlerrcxt, true); + + PG_RE_THROW(); } + PG_END_TRY(); + + xmlBufferFree(buf); + xmlFreeDoc(doc); - result = (text *) xmlBuffer_to_xmltype(buf); + pg_xml_done(xmlerrcxt, false); } - PG_CATCH(); + else if (format == XMLSERIALIZE_CANONICAL || format == XMLSERIALIZE_CANONICAL_WITH_COMMENTS) { - if (ctxt) - xmlSaveClose(ctxt); - if (buf) - xmlBufferFree(buf); - if (doc) - xmlFreeDoc(doc); + xmlChar *xmlbuf = NULL; + int nbytes; + int with_comments = 0; /* 0 = no xml comments (default) */ - pg_xml_done(xmlerrcxt, true); + PG_TRY(); + { + /* 1 = keeps xml comments */ + if (format == XMLSERIALIZE_CANONICAL_WITH_COMMENTS) + with_comments = 1; - PG_RE_THROW(); - } - PG_END_TRY(); + if (doc == NULL || escontext.error_occurred) + { + if (doc) + xmlFreeDoc(doc); + /* A soft error must be failure to conform to XMLOPTION_DOCUMENT */ + ereport(ERROR, + (errcode(ERRCODE_NOT_AN_XML_DOCUMENT), + errmsg("not an XML document"))); + } - xmlBufferFree(buf); - xmlFreeDoc(doc); + /* + * This dumps the canonicalized XML doc into the xmlChar* buffer. + * mode = 2 means the doc will be canonicalized using the C14N 1.1 standard. + */ + nbytes = xmlC14NDocDumpMemory(doc, NULL, 2, NULL, with_comments, &xmlbuf); - pg_xml_done(xmlerrcxt, false); + if(nbytes < 0 || escontext.error_occurred) + ereport(ERROR, + (errcode(ERRCODE_INTERNAL_ERROR), + errmsg("could not canonicalize the given XML document"))); + + result = cstring_to_text_with_len((const char *) xmlbuf, nbytes); + } + PG_CATCH(); + { + if (ctxt) + xmlSaveClose(ctxt); + if (xmlbuf) + xmlFree(xmlbuf); + if (doc) + xmlFreeDoc(doc); + + pg_xml_done(xmlerrcxt, true); + + PG_RE_THROW(); + } + PG_END_TRY(); + + xmlFreeDoc(doc); + xmlFree(xmlbuf); + + pg_xml_done(xmlerrcxt, false); + } + else + elog(ERROR,"invalid xmlserialize option"); return result; + #else NO_XML_SUPPORT(); return NULL; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 028588fb33..919d7158f4 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -841,6 +841,7 @@ typedef struct XmlSerialize Node *expr; TypeName *typeName; bool indent; /* [NO] INDENT */ + XmlSerializeFormat format; /* serialization format */ int location; /* token location, or -1 if unknown */ } XmlSerialize; diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h index 8fb5b4b919..2cdb1d970e 100644 --- a/src/include/nodes/primnodes.h +++ b/src/include/nodes/primnodes.h @@ -1474,6 +1474,14 @@ typedef enum XmlOptionType XMLOPTION_CONTENT } XmlOptionType; +typedef enum XmlSerializeFormat +{ + XMLSERIALIZE_INDENT, /* pretty-printed xml serialization */ + XMLSERIALIZE_CANONICAL, /* canonical form without xml comments */ + XMLSERIALIZE_CANONICAL_WITH_COMMENTS, /* canonical form with xml comments */ + XMLSERIALIZE_NO_FORMAT /* unformatted xml representation */ +} XmlSerializeFormat; + typedef struct XmlExpr { Expr xpr; @@ -1489,13 +1497,13 @@ typedef struct XmlExpr List *args; /* DOCUMENT or CONTENT */ XmlOptionType xmloption pg_node_attr(query_jumble_ignore); - /* INDENT option for XMLSERIALIZE */ - bool indent; /* target type/typmod for XMLSERIALIZE */ Oid type pg_node_attr(query_jumble_ignore); int32 typmod pg_node_attr(query_jumble_ignore); /* token location, or -1 if unknown */ int location; + /* serialization format: XMLCANONICAL, XMLCANONICAL_WITH_COMMENTS, XMLINDENT */ + XmlSerializeFormat format pg_node_attr(query_jumble_ignore); } XmlExpr; /* ---------------- diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 753e9ee174..8280f605d1 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -67,6 +67,7 @@ PG_KEYWORD("by", BY, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cache", CACHE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("call", CALL, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("called", CALLED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("canonical", CANONICAL, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cascade", CASCADE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("cascaded", CASCADED, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("case", CASE, RESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/utils/xml.h b/src/include/utils/xml.h index 224f6d75ff..b74f216148 100644 --- a/src/include/utils/xml.h +++ b/src/include/utils/xml.h @@ -78,7 +78,7 @@ extern xmltype *xmlpi(const char *target, text *arg, bool arg_is_null, bool *res extern xmltype *xmlroot(xmltype *data, text *version, int standalone); extern bool xml_is_document(xmltype *arg); extern text *xmltotext_with_options(xmltype *data, XmlOptionType xmloption_arg, - bool indent); + XmlSerializeFormat format); extern char *escape_xml(const char *str); extern char *map_sql_identifier_to_xml_name(const char *ident, bool fully_escaped, bool escape_period); diff --git a/src/test/regress/expected/xml.out b/src/test/regress/expected/xml.out index 398345ca67..83fbdc5223 100644 --- a/src/test/regress/expected/xml.out +++ b/src/test/regress/expected/xml.out @@ -672,6 +672,118 @@ SELECT xmlserialize(CONTENT '<foo><bar><val x="y">42</val></bar></foo>' AS text t (1 row) +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(DOCUMENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +ERROR: syntax error at or near "INDENT" at character 75 +SELECT xmlserialize(CONTENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +ERROR: syntax error at or near "INDENT" at character 74 +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ?column? ---------- diff --git a/src/test/regress/expected/xml_1.out b/src/test/regress/expected/xml_1.out index 63b779470f..481badaa84 100644 --- a/src/test/regress/expected/xml_1.out +++ b/src/test/regress/expected/xml_1.out @@ -443,6 +443,114 @@ ERROR: unsupported XML feature LINE 1: SELECT xmlserialize(CONTENT '<foo><bar><val x="y">42</val><... ^ DETAIL: This functionality requires the server to be built with libxml support. +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +ERROR: unsupported XML feature +LINE 2: (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + ^ +DETAIL: This functionality requires the server to be built with libxml support. +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +-------------- +(0 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- +(0 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: unsupported XML feature at character 30 +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: unsupported XML feature at character 29 +SELECT xmlserialize(DOCUMENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +ERROR: syntax error at or near "INDENT" at character 75 +SELECT xmlserialize(CONTENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +ERROR: syntax error at or near "INDENT" at character 74 +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ERROR: unsupported XML feature LINE 1: SELECT xml '<foo>bar</foo>' IS DOCUMENT; diff --git a/src/test/regress/expected/xml_2.out b/src/test/regress/expected/xml_2.out index 43c2558352..6dea6ca38d 100644 --- a/src/test/regress/expected/xml_2.out +++ b/src/test/regress/expected/xml_2.out @@ -652,6 +652,118 @@ SELECT xmlserialize(CONTENT '<foo><bar><val x="y">42</val></bar></foo>' AS text t (1 row) +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; + xmlserialize +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + <foo xmlns="http://example.org" xmlns:a="http://www.w3.org" xmlns:b="http://www.ietf.org" attr="I am" attr2="all" b:attr="sorted" a:attr="out"><bar xmlns="">42</bar><empty></empty><transcode>1</transcode><whitespace> 321 </whitespace><emptyns xmlns=""><emptyns_child></emptyns_child></emptyns><compute>value>"0" && value<"10" ?"valid":"error"</compute></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; + xmlserialize +--------------------------------------------------------------------- + <foo><bar><!-- important comment --><val x="y">42</val></bar></foo> +(1 row) + +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; + ?column? +---------- + t + t +(2 rows) + +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); + xmlserialize +-------------- + +(1 row) + +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +ERROR: invalid XML document +SELECT xmlserialize(DOCUMENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +ERROR: syntax error at or near "INDENT" at character 75 +SELECT xmlserialize(CONTENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +ERROR: syntax error at or near "INDENT" at character 74 +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; ?column? ---------- diff --git a/src/test/regress/sql/xml.sql b/src/test/regress/sql/xml.sql index a591eea2e5..a3eda9e84f 100644 --- a/src/test/regress/sql/xml.sql +++ b/src/test/regress/sql/xml.sql @@ -168,6 +168,69 @@ SELECT xmlserialize(CONTENT '<foo><bar></bar></foo>' AS text INDENT); -- 'no indent' = not using 'no indent' SELECT xmlserialize(DOCUMENT '<foo><bar><val x="y">42</val></bar></foo>' AS text) = xmlserialize(DOCUMENT '<foo><bar><val x="y">42</val></bar></foo>' AS text NO INDENT); SELECT xmlserialize(CONTENT '<foo><bar><val x="y">42</val></bar></foo>' AS text) = xmlserialize(CONTENT '<foo><bar><val x="y">42</val></bar></foo>' AS text NO INDENT); +-- xmlserialize: canonical +CREATE TABLE xmltest_serialize (id int, doc xml); +INSERT INTO xmltest_serialize VALUES + (1,'<?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE doc SYSTEM "doc.dtd" [ + <!ENTITY val "42"> + <!ATTLIST xyz attr CDATA "default"> + ]> + + <!-- attributes and namespces will be sorted --> + <foo a:attr="out" b:attr="sorted" attr2="all" attr="I am" + xmlns:b="http://www.ietf.org" + xmlns:a="http://www.w3.org" + xmlns="http://example.org"> + + <!-- Normalization of whitespace in start and end tags --> + <!-- Elimination of superfluous namespace declarations, as already declared in <foo> --> + <bar xmlns="" xmlns:a="http://www.w3.org" >&val;</bar > + + <!-- empty element will be converted to start-end tag pair --> + <empty/> + + <!-- text will be transcoded to UTF-8 --> + <transcode>1</transcode> + + <!-- default attribute will be added --> + <!-- whitespace inside tag will be preserved --> + <whitespace> 321 </whitespace> + + <!-- empty namespace will be removed of child tag --> + <emptyns xmlns="" > + <emptyns_child xmlns=""></emptyns_child> + </emptyns> + + <!-- CDATA section will be replaced by its value --> + <compute><![CDATA[value>"0" && value<"10" ?"valid":"error"]]></compute> + </foo> + <!-- comment outside doc -->'::xml), + (2,'<foo> + <bar> + <!-- important comment --> + <val x="y">42</val> + </bar> + </foo> '::xml); + +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; +SELECT xmlserialize(DOCUMENT doc AS text CANONICAL) = xmlserialize(DOCUMENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; +SELECT xmlserialize(CONTENT doc AS text CANONICAL) FROM xmltest_serialize WHERE id = 1; +SELECT xmlserialize(CONTENT doc AS text CANONICAL WITH COMMENTS) FROM xmltest_serialize WHERE id = 2; +SELECT xmlserialize(CONTENT doc AS text CANONICAL) = xmlserialize(CONTENT doc AS text CANONICAL WITH NO COMMENTS) FROM xmltest_serialize; +SELECT xmlserialize(DOCUMENT NULL AS text CANONICAL); +SELECT xmlserialize(CONTENT NULL AS text CANONICAL); +\set VERBOSITY terse +SELECT xmlserialize(DOCUMENT '' AS text CANONICAL); +SELECT xmlserialize(DOCUMENT ' ' AS text CANONICAL); +SELECT xmlserialize(DOCUMENT 'foo' AS text CANONICAL); +SELECT xmlserialize(CONTENT '' AS text CANONICAL); +SELECT xmlserialize(CONTENT ' ' AS text CANONICAL); +SELECT xmlserialize(CONTENT 'foo' AS text CANONICAL); +SELECT xmlserialize(DOCUMENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +SELECT xmlserialize(CONTENT '<foo><bar>73</bar></foo>' AS text CANONICAL INDENT); +\set VERBOSITY default SELECT xml '<foo>bar</foo>' IS DOCUMENT; SELECT xml '<foo>bar</foo><bar>foo</bar>' IS DOCUMENT; -- 2.25.1 ^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: [PATCH] Add CANONICAL option to xmlserialize 2023-03-05 21:00 Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-05 22:20 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-05 23:32 ` Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-06 10:50 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-06 13:19 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-14 07:49 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-17 09:46 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-17 12:30 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> @ 2023-09-14 11:54 ` Jim Jones <[email protected]> 2023-09-14 21:43 ` Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 0 siblings, 1 reply; 28+ messages in thread From: Jim Jones @ 2023-09-14 11:54 UTC (permalink / raw) To: Thomas Munro <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> The cfbot started complaining about this patch on "macOS - Ventura - Meson" 'Persistent worker failed to start the task: tart isolation failed: failed to create VM cloned from "ghcr.io/cirruslabs/macos-ventura-base:latest": tart command returned non-zero exit code: ""' Is this a problem in my code or in the CI itself? Thanks! Jim ^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: [PATCH] Add CANONICAL option to xmlserialize 2023-03-05 21:00 Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-05 22:20 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-05 23:32 ` Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-06 10:50 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-06 13:19 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-14 07:49 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-17 09:46 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-03-17 12:30 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> 2023-09-14 11:54 ` Re: [PATCH] Add CANONICAL option to xmlserialize Jim Jones <[email protected]> @ 2023-09-14 21:43 ` Thomas Munro <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Thomas Munro @ 2023-09-14 21:43 UTC (permalink / raw) To: Jim Jones <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> On Thu, Sep 14, 2023 at 11:54 PM Jim Jones <[email protected]> wrote: > The cfbot started complaining about this patch on "macOS - Ventura - Meson" > > 'Persistent worker failed to start the task: tart isolation failed: failed to create VM cloned from "ghcr.io/cirruslabs/macos-ventura-base:latest": tart command returned non-zero exit code: ""' > > Is this a problem in my code or in the CI itself? There was a temporary glitch on one of the new Mac CI runner machines that caused a few tests to fail like that, but it was fixed so that should turn red again later today. ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v10 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 5df3b5ca46..404de0595e 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --3o7pc6dfau5a5hry Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v10-0006-table_scan_bitmap_next_block-returns-lossy-or-ex.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v11 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 5df3b5ca46..404de0595e 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --owzzsiozz6hgpp7e Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v11-0006-table_scan_bitmap_next_block-returns-lossy-or-ex.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v6 05/14] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 6a34f10646a..61ffc8d0ee8 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -228,7 +228,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); if (!table_scan_bitmap_next_block(scan, tbmres)) { @@ -337,7 +337,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -356,7 +356,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --w4wcjcocxsm37usi Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6-0006-EXPLAIN-Bitmap-table-scan-also-count-no-visible-t.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v8 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 5df3b5ca46..404de0595e 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --xqq4defy3uncu6k6 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v8-0006-table_scan_bitmap_next_block-returns-lossy-or-exa.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v9 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 5df3b5ca46..404de0595e 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --7mdtsjmrzitrgzgx Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v9-0006-table_scan_bitmap_next_block-returns-lossy-or-exa.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v7 05/13] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index dc8c312cafb..a6b98fa12a1 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --kqqpqghcwbcc3dt5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0006-table_scan_bitmap_next_block-returns-lossy-or-exa.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v7 05/13] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index dc8c312cafb..a6b98fa12a1 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --kqqpqghcwbcc3dt5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0006-table_scan_bitmap_next_block-returns-lossy-or-exa.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v9 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 5df3b5ca46..404de0595e 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --7mdtsjmrzitrgzgx Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v9-0006-table_scan_bitmap_next_block-returns-lossy-or-exa.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v9 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 5df3b5ca46..404de0595e 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --7mdtsjmrzitrgzgx Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v9-0006-table_scan_bitmap_next_block-returns-lossy-or-exa.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v7 05/13] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index dc8c312cafb..a6b98fa12a1 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --kqqpqghcwbcc3dt5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0006-table_scan_bitmap_next_block-returns-lossy-or-exa.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v10 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 5df3b5ca46..404de0595e 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --3o7pc6dfau5a5hry Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v10-0006-table_scan_bitmap_next_block-returns-lossy-or-ex.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v6 05/14] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 6a34f10646a..61ffc8d0ee8 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -228,7 +228,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); if (!table_scan_bitmap_next_block(scan, tbmres)) { @@ -337,7 +337,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -356,7 +356,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --w4wcjcocxsm37usi Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6-0006-EXPLAIN-Bitmap-table-scan-also-count-no-visible-t.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v8 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 5df3b5ca46..404de0595e 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --xqq4defy3uncu6k6 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v8-0006-table_scan_bitmap_next_block-returns-lossy-or-exa.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v6 05/14] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 6a34f10646a..61ffc8d0ee8 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -228,7 +228,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); if (!table_scan_bitmap_next_block(scan, tbmres)) { @@ -337,7 +337,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -356,7 +356,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --w4wcjcocxsm37usi Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6-0006-EXPLAIN-Bitmap-table-scan-also-count-no-visible-t.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v8 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 5df3b5ca46..404de0595e 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --xqq4defy3uncu6k6 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v8-0006-table_scan_bitmap_next_block-returns-lossy-or-exa.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v12 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 5df3b5ca46d..404de0595ec 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --6jpz2j246qmht4bt Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v12-0006-table_scan_bitmap_next_block-returns-lossy-or-ex.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v10 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 5df3b5ca46..404de0595e 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --3o7pc6dfau5a5hry Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v10-0006-table_scan_bitmap_next_block-returns-lossy-or-ex.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v11 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber @ 2024-02-13 00:04 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 28+ messages in thread From: Melanie Plageman @ 2024-02-13 00:04 UTC (permalink / raw) BitmapAdjustPrefetchIterator() only used the blockno member of the passed in TBMIterateResult to ensure that the prefetch iterator and regular iterator stay in sync. Pass it the BlockNumber only. This will allow us to move away from using the TBMIterateResult outside of table AM specific code. --- src/backend/executor/nodeBitmapHeapscan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 5df3b5ca46..404de0595e 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -52,7 +52,7 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres); + BlockNumber blockno); static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); static inline void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); @@ -230,7 +230,7 @@ BitmapHeapNext(BitmapHeapScanState *node) break; } - BitmapAdjustPrefetchIterator(node, tbmres); + BitmapAdjustPrefetchIterator(node, tbmres->blockno); valid = table_scan_bitmap_next_block(scan, tbmres); @@ -341,7 +341,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) */ static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, - TBMIterateResult *tbmres) + BlockNumber blockno) { #ifdef USE_PREFETCH ParallelBitmapHeapState *pstate = node->pstate; @@ -360,7 +360,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node, /* Do not let the prefetch iterator get behind the main one */ TBMIterateResult *tbmpre = tbm_iterate(prefetch_iterator); - if (tbmpre == NULL || tbmpre->blockno != tbmres->blockno) + if (tbmpre == NULL || tbmpre->blockno != blockno) elog(ERROR, "prefetch and main iterators are out of sync"); } return; -- 2.40.1 --owzzsiozz6hgpp7e Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v11-0006-table_scan_bitmap_next_block-returns-lossy-or-ex.patch" ^ permalink raw reply [nested|flat] 28+ messages in thread
end of thread, other threads:[~2024-02-13 00:04 UTC | newest] Thread overview: 28+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2023-03-05 21:00 Re: [PATCH] Add CANONICAL option to xmlserialize Thomas Munro <[email protected]> 2023-03-05 22:20 ` Jim Jones <[email protected]> 2023-03-05 23:32 ` Thomas Munro <[email protected]> 2023-03-06 10:50 ` Jim Jones <[email protected]> 2023-03-06 13:19 ` Jim Jones <[email protected]> 2023-03-14 07:49 ` Jim Jones <[email protected]> 2023-03-17 09:46 ` Jim Jones <[email protected]> 2023-03-17 12:30 ` Jim Jones <[email protected]> 2023-09-14 11:54 ` Jim Jones <[email protected]> 2023-09-14 21:43 ` Thomas Munro <[email protected]> 2024-02-13 00:04 [PATCH v10 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v11 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v6 05/14] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v8 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v9 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v7 05/13] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v7 05/13] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v9 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v9 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v7 05/13] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v10 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v6 05/14] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v8 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v6 05/14] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v8 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v12 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v10 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]> 2024-02-13 00:04 [PATCH v11 05/17] Update BitmapAdjustPrefetchIterator parameter type to BlockNumber Melanie Plageman <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox