1 This is a TODO file for XSL-T 2.0 support.

     2 

     3 - LHF:

     4     * Warning bug, last parameter is always whined about.

     5     * Box in comment/PI/text/ws(?) handling -- pending Matthias

     6     * type036 -- namespace on top element isn't copied

     7     * XTDE0865

     8     * Attend XSLTTokenizer::isXSLT()

     9     * Remove redundant set() calls in setFocusHelper().

    10 

    11 - Missing features:

    12     General                                     Priority

    13     ---------------------

    14     * 1.0 QXmlQuery::evaluateTo(QIODevice *)    P1 DONE

    15     * 1.0 Test suite integration                P1 DONE

    16     * 1.0 xsl:key                               P1

    17     * 1.0 fn:key()                              P1

    18     * 1.0 2.0 Compatibility mode                P1

    19     * 1.0 Regular parameters in templates       P1

    20     * 1.0 xsl:include                           P1

    21     * 1.0 xsl:copy-of                           P1

    22     * 1.0 xsl:copy                              P1

    23     * 1.0 xsl:import                            P1

    24     * 1.0 fn:format-number                      P1

    25     * 1.0 xsl:message                           P2

    26     * 1.0 fn:current()                          P1 DONE

    27     * 2.0 fn:type-available()                   P3 DONE

    28     * 2.0 xsl:use-when                          P3

    29     * 2.0 fn:unparsed-entity-uri()              P3

    30     * 2.0 fn:unparsed-entity-public-id()        P3

    31     * 2.0 Tunnel Parameters                     P3

    32     * 2.0 xsl:attribute-set                     P3

    33     * 1.0 xsl:decimal-format                    P2

    34     * 1.0 xmlpatterns: initial template         P1 DONE

    35     * 1.0 xsl:number                            P1

    36     * 1.0 Complete handling of xsl:sort         P2

    37     * 2.0 Grouping

    38         - fn:current-group()

    39         - fn:grouping-key()

    40         - xsl:for-each-group()

    41     * 2.0 Regexp

    42         - xsl:analyze-string

    43         - xsl:matching-substring

    44         - xsl:non-matching-substring

    45         - fn:regex-group()

    46     * 2.0 Date & Time formatting

    47         - fn:format-dateTime()

    48         - fn:format-date()

    49         - fn:format-time()

    50 

    51     Serialization & Output:

    52     ----------------------

    53     * 1.0 xsl:output

    54     --- Tie together serialization. Should we add

    55         QXmlQuery::evaluateTo(QIODevice 1.0 const) ?

    56     * 2.0 xsl:character-maps

    57     * 2.0 xsl:character-map

    58     * 2.0 xsl:result-document

    59     --- Should the "default output" be handle with xsl:result-document? Would

    60         depend on compilation.

    61 

    62 Optimizations:

    63     * Remove adjacent text node constructors

    64     * Remove string-join when first arg's static cardinality is not more than one

    65     * Remove string-join when the second arg is statically known to be the empty string.

    66     * Remove string-join when the second arg is a single space and the parent is a text node ctor.

    67     * Rewrite to operand if operands are one. What about type conversions?

    68     * Replace lookups with xml:id with calls on id().

    69     * Recognize that a/(b, c) is equal to a/(b | c). The later does selection and node sorting in one step.

    70     * Remove LetClause which has empty sequence as return clause, or no variable dependencies at all.

    71     * Do a mega test for rewriting /patterns/:

    72                     "node() | element()" => element()

    73                     "comment() | node()" => comment()

    74 

    75       and so forth. This sometimes happens in poorly written patterns. How does

    76       this rewrite affect priority calculation?

    77 

    78 Tests:

    79     * xml:id

    80         - Come on, the stuff needs to be reorganized xml:id.

    81         - Read in xml:id document with whitespace in attrs, write the doc out. Attrs should be normalized.

    82         - Do lookups of IDs with xml:id attrs containing whitespace.

    83 

    84     * current()

    85         - Use current() inside each instruction

    86         - In a template pattern

    87         - Several invocations: current()/current()/current()

    88 

    89 

    90     * Diagnosticsts:

    91         - See http://www.w3.org/Bugs/Public/show_bug.cgi?id=5643 . Comments

    92           should be taken into account when comparing. This suggests that we

    93           don't have any test which produces a document with XML comments.

    94 

    95     * element-available()

    96         - Review the tests.

    97         - Try using declarations in XSL-T, should return false

    98         - Use xsl:variable(both instr and decl)

    99         - invoke with all the XSL-T instructions.

   100         - Should return false for when, otherwise, matching-substring, non-matching-substring, etc?

   101         - Supply the namespace in the name via the default namespace, no prefix.

   102 

   103     * unparsed-text()

   104         - Load an empty file

   105         - Use a fragment in the URI

   106         - Use an invalid URI

   107         - Use device bindings and a QRC to ensure that we're not using a generic

   108           network manager.

   109         - Duplicate all the network tests. Same as for doc()

   110 

   111     * unparsed-text-available()

   112         - Same as for unparsed-text()

   113 

   114     * Sequence constructor that contains only:

   115         - XML comment

   116         - whitespace text node

   117         - processing instruction

   118         - a mix of the three

   119 

   120     * xsl:function

   121         - Ensure that it's not it's not in scope for use-when.

   122         - xsl:function/xsl:param: use processing instructions, whitespace and comments as child: should be stripped

   123         - Use <xsl:function/> : @name missing.

   124         - Don't strip ws, and have ws between two xsl:param, and between xsl:function and xsl:param.

   125         - Use xsl:function with no body.

   126         - use xsl:param/@tunnel = no

   127         - use xsl:param/@tunnel = yes

   128         - use an invalid value for xsl:param/@tunnel = yes

   129         - Have a non-WS text node in xsl:function/xsl:param/

   130         - Have a WS text node in xsl:function/xsl:param/

   131         - Have a WS text node in xsl:function/xsl:param/ while preserving WS.

   132         - use a comment as child of xsl:param

   133         - use a PI as child of xsl:param

   134         - XTSE0770 with import precedence and all that.

   135         - have two identical function in the stylesheet. The last has override=no. Should still report XTSE0770.

   136         - have @override with invalid value.

   137         - have whitespace inside xsl:param with different strip modes.

   138         - Have @select => error

   139         - Have body => error

   140         - call current() inside body. XPDY0002?

   141 

   142     * Does xml:base/StaticBaseURI and StaticCompatiblityStore prevent proper

   143           type checking due to expectedOperandTypes() returns item()*?

   144 

   145     * xsl:template/xsl:param

   146         - Have @required=yes, and have @select => error

   147         - Have @required=yes, and have body => error

   148         - Have a variable reference in a template after another, which has

   149           param, to ensure they aren't in scope.

   150 

   151     * xsl:template/@match

   152         - Have a pattern with unions, and have a body which relies on its

   153           static type.

   154 

   155     * @version:

   156         Have @version on *all* attributes.

   157 

   158     * xsl:call-template

   159         - Have a variable reference just after a xsl:call-template which has

   160           with-param, to ensure they aren't in scope.

   161         - Have an xsl:with-param which isn't used in the template. Error?

   162         - Have an xsl:with-param that has a type error.

   163         - an xsl:with-param is not in scope for the next one. Test this => error.

   164         - Have a call:template, whose with-param computes its value by calling

   165           another template, while using an with-param too.

   166 

   167     * XQuery:

   168         - DONE Ensure namespace {expr} {expr} is flagged as invalid

   169         - Use all XSL-T functions: error. Or we do that already?

   170         - Ensure order by collation 1 + 1 is an error

   171         - Ensure order by collation {1 + 1} is an error

   172 

   173     * document()

   174         - Basic node deduplication, no test exists for that.

   175 

   176     * xsl:perform-sort

   177         - Have no xsl:sort. Error. Must be at least one.

   178         - have xsl:sort with invalid value.

   179         - sort atomic values.

   180         - Trigger "The stable attribute is permitted only on the first xsl:sort  element within a sort key specification"

   181         - have xsl:sort with no select and no seq ctor.

   182         - trigger the delegated queueing. All instructions inside.. xsl:sort?

   183         - have multiple sort statements, with the last being <xsl:sort/> only.

   184         - have WS between xsl:sort that is not ignorable.

   185         - Use a variable reference whose name is equal to our synthetic name. This should be XPST0008, but probably isn't.

   186         - Have an invalid value in xsl:sort/order. Use space

   187         - have xsl:sort return numbers, but data-type specify string.

   188         - have an AVT in xsl:sort/@lang

   189         - have an AVT in xsl:sort/@case-order

   190         - have an AVT in xsl:sort/@data-type

   191         - have an AVT in xsl:sort/@stable

   192         - Have mixed result, and hence incorrectly trigger  XPTY0018 which the code currently raise.

   193         - Depend on the context position inside xsl:sort, when being child of

   194           perform-sort. Currently we create singleton focuses(I think), while

   195           we want the focus to be over the whole input sequence, not on indivual items.

   196         - Have <xsl:perform-sort select="valid-expr"/>: xsl:sort is missing

   197         - Use current() in the xsl:sort and the body, to ensure the right scope is picked up

   198 

   199     * xsl:copy-of

   200         - Have a text node. It's not allowed.

   201         - Have PIs, comments, and ignorable whitespace as children. Sigh.

   202 

   203     * xsl:namespace

   204         - Use xsl:fallback.

   205         - Use xsl:namespace inside xsl:variable and introspec the result in various

   206           ways. This is a big area, we don't have namespace nodes in XQuery. Yes, calling evaluateSingleton() will probably crash.

   207         - Use no select and no body, error: XTSE0910

   208         - Have name expression evaluate to the empty sequence.

   209 

   210     * Sequence ctor that:

   211         - Has invalid element in XSL namespace. E.g, xsl:foo

   212 

   213     * xsl:import

   214         - Have element as child as xsl:import: disallowed.

   215         - Have text as child as xsl:import: disallowed.

   216         - Have PIs and comments as child as xsl:import: allowed.

   217 

   218     * xsl:include

   219         - Have element as child as xsl:include: disallowed.

   220         - Have text as child as xsl:include: disallowed.

   221         - Have PIs and comments as child as xsl:include: allowed.

   222 

   223     * xsl:strip-space

   224         - Have PIs, comments, whitespace as child.

   225 

   226     * xsl:element

   227         - Extract EBV from result.

   228         - Use space in validation element.

   229 

   230     * xsl:perform-sort

   231         - Have PIs and comments in between xsl:sort elements.

   232 

   233     * xml:space

   234         - We never pop our stack. Fix the bug, and ensure we have tests for it.

   235 

   236     * fn:unparsed-entity-uri

   237         - Check type of return value

   238         - Do basic unparsed-entity-uri("does-not-exist")

   239 

   240     * fn:unparsed-entity-public-id

   241         - Do basic unparsed-entity-uri("does-not-exist"), two permutations, check the spec

   242 

   243     * xsl:element

   244         - Use disallowed attribute: select

   245         - use unknown type in @type

   246         - Use @namespace, but be not in the lexical space of xs:anyURI

   247         - use disallowed enumeration in @validation

   248         - have a name expression that evaluates to a xs:QName value as opposed to a string.

   249         - have a name expression that evaluates to a xs:QName value as opposed to a string. but

   250           also have the namespace attribute

   251 

   252     * xsl:attribute

   253         - Use disallowed attribute: match

   254         - use unknown type in @type

   255         - Use @namespace, but be not in the lexical space of xs:anyURI

   256         - use disallowed enumeration in @validation

   257         - have a name expression that evaluates to a xs:QName value as opposed to a string.

   258         - have a name expression that evaluates to a xs:QName value as opposed to a string. but

   259           also have the namespace attribute

   260 

   261     * xsl:template

   262         - Use the union keyword, it's forbidden, only "|" is allowed

   263         - Use an expression other than Literal and VarRef in KeyValue[8]

   264         - use a function other than key().

   265         - have a declaration that only can apperar as a child of xsl:stylesheet.

   266         - Have an element in the XSL-T namespace, but which is invalid, e.g "bar"

   267         - Use an axis other than child or attribute in pattern.

   268         - Have a template that no no match and no name attribute.,  XTSE0500

   269         - use child::document-node() in pattern

   270         - use @foo/child in pattern

   271         - apply templates to parentless attributes.

   272         - Have 3e3 in @priority

   273         - Have a @match with more than two alternatives, e.g "a | b | c", and have them all actually matching.

   274         - Use an XML name in the mode so we trigger

   275           NCNameConstructor::validateTargetName()

   276         - A template which only has a non-WS text node.

   277         - A template with param, followed by text node.

   278 

   279     * Simplified stylesheet

   280         - Use @version attribute only on doc element. Should fail, since @{XSL-T]version must be present

   281 

   282     * fn:current()

   283         - Have <xsl:value-of select="current()"/>

   284 

   285     * xsl:variable have a variable reference appearing before its global declaration, and then somehow trigger recursion.

   286     * xsl:choose

   287         - elements/nodes intermixed with xsl:choose/xsl:when

   288         - invalid attribute on xsl:choose

   289         - invalid attribute on xsl:when

   290         - invalid attribute on xsl:otherwise

   291         - invalid attribute on xsl:if

   292         - invalid attribute on xsl:template

   293         - invalid attribute on xsl:stylesheet

   294         - invalid attribute on xsl:transform

   295         - xsl:otherwise in the middle between xsl:when elements.

   296         - use namespace declarations on xsl:when

   297         - use namespace declarations on xsl:otherwise

   298         - use namespace declarations on xsl:choose

   299 

   300     * Namespaces:

   301         - Have:

   302             <xsl:sequence xmlns:bar="http://example.com/" select="1"/>

   303             <xsl:sequence select="bar:foo()"/>

   304 

   305     * XPath

   306         - For each XQuery-specific expression, add a test using that expression:

   307             - typeswitch

   308             - let

   309             - validate

   310             - extension expression

   311             - unordered

   312             - ordered

   313             - for

   314             - computed text node constructor

   315             - computed attribute constructor

   316             - computed comment constructor

   317             - computed PI constructor

   318             - computed element constructor

   319             - computed document constructor

   320             - direct element constructor

   321             - direct comment constructor

   322             - direct PI constructor

   323             - all declarations

   324 

   325         - Use all the predefined prefixes in XQuery; non are in XSL-T.

   326 

   327     * xsl:when

   328         - Use xml:space on it

   329 

   330     * xsl:otherwise

   331         - Use xml:space on it

   332 

   333     * xsl:version

   334         - Use letters, XTSE0110

   335         - Use a float: 2e3, XTSE0110

   336         - Use a weird number, 2.00000001

   337 

   338     * xsl:document

   339         - use disallowed attribute: select.

   340         - use unknown type in @type

   341         - use disallowed enumeration in @validation

   342         - What happens if the type in @type is unknown?

   343         - Use xml:base attr and check URI.

   344 

   345     * xsl:sequence

   346         - use match attribute

   347 

   348     * xsl:stylesheet

   349         - Use @xsl:default-collation on xsl:stylesheet. Shouldn't have any effect. Or?

   350         - Use an XSL-T instruction as child -- invalid.

   351         - Have an element in the XSL-T namespace, but which is invalid, e.g "foo"

   352         - Have xsl:default-collation="http://example.com/" on xsl:stylesheet

   353         - Use prefix local: in say a function name. Not allowed.

   354         - Use comments after document element.

   355         - XTSE0010: <xsl:invalid version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"/>

   356         - Change the version with @xsl:version on all elements that we have.

   357 

   358     * Patterns.

   359         - Basic */* test:

   360             <xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"><xsl:template match="*/*"><xsl:sequence select="'MATCH'"/></xsl:template>

   361 

   362 

   363 </xsl:stylesheet>

   364 

   365         - Basic a/b test:

   366             <xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"><xsl:template match="a/b"><xsl:sequence select="'MATCH'"/></xsl:template>

   367 

   368 

   369 </xsl:stylesheet>

   370     * xsl:strip-whitespace

   371         - Use a namespace prefix which is not unboudn

   372         - have a syntax error in one of the node tests

   373 

   374     * xsl:preserve-whitespace

   375         - Use a namespace prefix which is not unboudn

   376         - have a syntax error in one of the node tests

   377 

   378     * xsl:value-of

   379         - select attribute, and comment in body(error, XTSE0870)

   380         - select attribute, and processing instruction in body(error, XTSE0870)

   381         - select attribute, and CCDATA in body(error, XTSE0870)

   382         - select attribute, and element in body(error, XTSE0870)

   383         - use xsl:sequence in body. Default separator should be none.

   384         - use match attribute

   385         - use double apostrophes/quotes. How are they dealt with?

   386 

   387     * xsl:apply-templates

   388         - use match attribute

   389         - apply in a mode for which no templates are declared

   390         - apply in a mode which is mispelled for another.

   391         - Have: <xsl:apply-templates select="id('id2')/a | id('id5')"/>

   392           We CRASH

   393 

   394     * xsl:for-each

   395         - No body: <xsl:for-each select="abc"/>

   396         - No select attribute: <xsl:for-each>text</xsl:for-each>

   397         - Have mixed result, and hence incorrectly trigger  XPTY0018 which the code currently raise.

   398         - Have:

   399             <xsl:for-each select="1, 'asd'">

   400                 <xsl:sequence select="."/>

   401             </xsl:for-each>

   402 

   403     * xsl:variable

   404         - Test that function conversion rules are invoked

   405         - For what is an xsl:variable in scope? Where does the spec state it? Test

   406           that it is not in scope where applicable.

   407         - Have: <variable name="a" select=""/>

   408 

   409     * xsl:text

   410         - count the result of a template that has text node(non-ws),

   411           xsl:text(content), xsl:content(zero content), text node(non-ws

   412         - Have an element inside xsl:text: XTSE0010.

   413         - Use comments and PIs intermixed with text inside.

   414 

   415     * xsl:for-each

   416         - use match attribute

   417         - What should this produce? Saxon produces "123" but with xsl:text removed, "1 2 3".

   418                 <xsl:template match="/">

   419                     <xsl:for-each select="1 to 3">

   420                         <xsl:sequence select="."/>

   421                         <xsl:text></xsl:text>

   422                     </xsl:for-each>

   423                 </xsl:template>

   424 

   425     * xsl:if

   426         - Have <xsl:if test="">body</xsl:if>. Error

   427         - Have <xsl:if test="valid-test"/>. That is, empty body.

   428 

   429     * xsl:sequence

   430         - select attribute missing: <xsl:sequence/>

   431         - content other than xsl:fallback, e.g text node.

   432         - How do we sort?

   433 

   434     * for every state for XSL-T parsing:

   435         - Use invalid element that is in the XSL-T namespace.

   436 

   437     * In all cases expressions are queued/generated:

   438         - Trigger expression precedence bugs, due to lack of paranteses.

   439 

   440     * Use xml:space in stylsheeet that doesn't have value preserve nor default.

   441     * For each case we have while(!reader.atEnd()):

   442         - test that triggers parser error and that we detect it properly.

   443 

   444     * for every element that allows text:

   445         * Use CDATA. QXmlStreamReader distinguishes between the two. text before and after.:wa

   446         * Use XML Comments and split up text nodes.

   447 

   448     * Patterns:

   449         * Ensure node() doesn't match document nodes().

   450         * "//" is an invalid pattern

   451         * Is there some expression which has no children? findAxisStep()

   452         * Use @*/asdo

   453         * XPST0003: key(concat("abc", "def"), "abc")

   454         * XPST0003: id(concat("abc", "def"))

   455         * XPST0003: concat('abc', 'def') // WILL CRASH

   456         * XPST0003: unknownFunction()

   457         * Use double: key("abc", 3e3)

   458         * Use three argument key() in pattern.

   459 

   460     * Simplified stylsheet modules:

   461         * Omit the xsl:version attribute. XTSE0010

   462 

   463     * type-available()

   464         * We have no tests at all?

   465 

   466     * have xml:base on the following elements and check them with

   467       static-base-uri():

   468         - all instructions

   469         - all declarations, eg:

   470         - xsl:choose, xsl:choice, xsl:otherwise

   471         - xsl:template

   472         - xsl:function

   473         - etc

   474 

   475     Embedded stylesheet modules

   476         - Verify that we don't choke on xsl attrs with invalid attributes outside;

   477           "In an embedded stylesheet module, standard attributes appearing on

   478           ancestors of the outermost element of the stylesheet module have no effect."

   479 

   480     Parsing:

   481         - Error message for:

   482                 <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

   483                     <xsl:template match="/">

   484                         <e/>

   485                 </xsl:stylesheet>

   486         - Use the document "<do/" as focus.

   487         - Write a test for each call to checkForParseError();

   488 

   489 

   490     function-available:

   491         - Generally low coverage it seems.

   492 

   493 

   494 <xsl:template match="/"/>                   <!-- t0 -->

   495 <xsl:template match="*"/>                   <!-- t1 -->

   496 <xsl:template match="asd"/>                 <!-- t2 -->

   497 <xsl:template match="comment()"/>           <!-- t3 -->

   498 <xsl:template match="a/b"/>                 <!-- t4 -->

   499 

   500 <xsl:apply-templates select="*"/>

   501 

   502 

   503 *(

   504 ((/)/call-template(t0))

   505 (*/call-template(t1))

   506 (element(asd)/call-template(t2))

   507 (comment()/call-template(t3))

   508 (a/b/call-template(t3))

   509 )

   510 

   511 ==>

   512 

   513 */typeswitch(.)

   514   case $g0 as document-root()   return call-template(t0)

   515   case $g0 as element()         return call-template(t1)

   516   case $g0 as element(asd)      return call-template(t2)

   517   case $g0 as comment()         return (call-template(t3)

   518   case $g0 as a/b               return (call-template(t4)

   519 

   520 

   521 Patterns are used in:

   522     - xsl:for-each-group/@group-starting-with

   523     - xsl:key/@match

   524     - xsl:number/(@count, @from)

   525     - xsl:template/@match

   526 

   527 

   528 

   529 c/b

   530 =>

   531 child-or-self::element(b)[parent::element(c)]

   532 

   533 c/b/a

   534 =>

   535 child-or-self::element(a)[parent::element(b)[parent::element(c)]]

   536 

   537 d/c/b/a

   538 =>

   539 child-or-self::element(a)[parent::element(b)[parent::element(c)[parent::element(d)]]]

   540 

   541 

   542 

   543 -----------------------------------

   544     <xsl:apply-templates select="foo"/>

   545                    =>

   546 

   547     child::element(foo) map apply-template(#default)

   548 -----------------------------------

   549 

   550 -----------------------------------

   551     <xsl:apply-templates mode="yo" select="foo">

   552         <xsl:sort select="@bar"/>

   553     </xsl:apply-templates>

   554                    =>

   555 

   556     let $g0 := for $g1 in child::element(foo)

   557                order by @bar

   558                return $g1

   559     return apply-template(yo)

   560 -----------------------------------

   561 

   562 -----------------------------------

   563 <xsl:perform-sort select="$in">

   564     <xsl:sort select="@sortKey"/>

   565 </xsl:perform-sort>

   566 

   567         =>

   568 

   569 sort $in/order by @sortKey

   570 -----------------------------------

   571 

   572 

   573 -----------

   574 

   575 John Snelson of Oracle Berkeley DB XML & XQilla writes in private mail:

   576 

   577 	I'd had the same thought myself, too - you must be reading my mind ;-)

   578 

   579 What is he referring to?

   580 

   581 If one spends some time on fancy diagrams, QtXmlPatterns[LINK]'s, the XQuery

   582 engine that shipped in Qt 4.4, architecture looks like this:

   583 

   584 

   585 Recently I've started implementing XSL-T 2.0(hopefully to be done for Qt 4.5)

   586 and the whole approach to this is modifying the existing codebase as follows:

   587 

   588 

   589 

   590 

   591 

   592 

   593 Put differently, when QtXmlPatterns is dealing with XSL-T stylesheets, it

   594 replaces the XQuery tokenizer with a tokenizer which translates the

   595 stylesheet into XQuery tokens, that is consumed by the existing XQuery

   596 parser, extended with both grammar non-terminals and tokens to accomodate the

   597 XSL-T features that XQuery doesn't have. In compiler terms, it can be seen as

   598 an "extreme" frontend. Not only is the same intermediate representation used,

   599 the grammar is too.

   600 

   601 What is the point of this?

   602 

   603 The functional overlaps XQuery, XSL-T and others as well have is of course

   604 widely established. Even the specifications are at times generated from the

   605 same source documents, and that implementations subsequently modularize code

   606 is of course second nature to any engineer, and seen to some degree or

   607 another in contemporary implementations. Typically this happens in a

   608 traditional fashion, classes are re-used, their functionality widened to

   609 cover both/more languages. However, I believe doing it directly on the

   610 grammar level introduce several advantages.

   611 

   612 The parser is based on Bison and since it's not running in the experimental

   613 pull mode, it uninterruptedly calls the tokenizer. The tokenizer, class

   614 XSLTTokenizer, in turns calls an pull-based XML parser: QXmlStreamReader.

   615 What often complicate in ocassions like this is who that gets the right to

   616 call who, and who gets the convenience of tracking state in a natural way

   617 through a call stack.

   618 

   619 XSLTTokenizer is conveniently implemented: as it encounters declarations and

   620 instructions in the stylsheet, it recursively descends in the XSL-T grammar

   621 through its own functions, adding tokens to a queue, which is delivered to

   622 the parser when asked -- and when the queue is empty it resumes queuing

   623 tokens. The tokenizer is fairly crude, it queues up tokens for instructions

   624 uninterrupted, and only have states between declarations. Hence,

   625 XSLTTokenizer queues up tokens for each template and function body, but

   626 enters "delivery mode" inbetween. This of course periodically breaks

   627 streaming since it's buffering up tokens, but considering that the memory

   628 usage for tokens is low and that a finer granularity for states(say, being

   629 able to pop the stacks when being inbetween two xsl:when elements) requires a

   630 significant effort, this is fine until proven otherwise.

   631 

   632 

   633 Advantages

   634 ---------------

   635 discuss analysis.

   636 

   637 

   638 XSLTTokenizer rewrite XSL-T to XQuery as follows:'

   639 

   640 Instructions

   641 -------------

   642 xsl:if An if/then/else expression whose else branch is the empty sequence

   643 

   644 xsl:choose: again, a nesting of if/then/else expressions

   645 

   646 xsl:value-of: a computed text node constructor. Its body contains a call to

   647 string-join() involving the separator attribute

   648 

   649 xsl:variable: a let/return binding. Since XSL-T is statement-like in its

   650 sequence constructors, parantheses are used to ensure the variable binding is

   651 in-scope for all subsequent statements.

   652 

   653 for-each: it is the iteration/mapping mechanism XQuery fails to supply,

   654 despite path steps and the FLWOR machinery. for-each iterates using a

   655 focus(which for doesn't, but paths do), but can do so over atomic values and

   656 unconditionally without sorting the result by document order(which paths

   657 can't/doesn't, but for do). For implementations that normalize paths into for

   658 loops as the formal semantics do, the approach is straight forward. In

   659 QtXmlPatterns' case, a synthetic token is queued which signals to create

   660 a "relaxed" path expression which skips halting on atomic values in its

   661 operands(XPTY0019) and also skips node sorting.

   662 

   663 All "direct" node constructors, like <myElement/>, and "computed" node

   664 constructors, like xsl:element, are all rewritten into the corresponding

   665 XQuery computed node constructors. In some cases direct node constructors

   666 could have been used, but in anycase the IR yielded is the same, and that

   667 computed constructors happen to use less tokens.

   668 

   669 A particular case is xsl:namespace, an instruction which doesn't have any

   670 corresponding expression in XQuery. In the case of QtXmlPatterns, the code

   671 obvious already have a notion of "create this namespace on this element", and

   672 an AST node was trivially added for fetching the namespace components

   673 computationally. However, the introduction of xsl:namespace in an XQuery

   674 implementation is not to be taken lightly wrt. to for instance testing, since

   675 it introduces a new node type.

   676 

   677 perform-sort: surprisingly this expression of all complicate matters, for the

   678 simple reason that its operands occur in the opposite order compared to

   679 XQuery when the input sequence is supplied through a sequence constructor,

   680 hence breaking the streamed approach. XSLTokenizer solves this by

   681 buffer: the attributes of the xsl:perform-sort elements are stored,

   682 the xsl:sort elements queued onto a temporary queue, and subsequently is

   683 either the select attribute or the sequence constructor queued, and the

   684 tokens for xsl:sort appended afterwards. This complicated code greatly, since

   685 XSLTokenizer had to be able to "move around" sequences of tokens.

   686 

   687 In addition perform-sort has the same problem as for-each, the iteration

   688 mechanism falls inbetween paths and the for loop. The focus for perform-sort

   689 is also the focus for the sequence constructor and the select attribute, but

   690 the focus for the xsl:sort elements is the initial sequence. This is

   691 approached by having a for loop, and where the expression in each order by

   692 clause has a relaxed path expression whose left operand is a variable

   693 reference to what the for loop bound.

   694 TODO Doesn't work. Focus size wrong.

   695 

   696 This is an approach that implementations of the "second generation" of the

   697 technologies can take. The bif difference is that XSL-T 2.0 doesn't have the

   698 restrictions of 1.0, more evident in XQuery's syntax.

   699 

   700 xsl:sort XSL-T is much more dynamic than XQuery through the use of templates,

   701 but also

   702 because more decisions can be taken at runtime through all attribute value

   703 templates. xsl:sort is surely a good example of this with its AVTs for

   704 language, order, collation, stability and what not. XQuery's order by stands

   705 in strong contrast, which has these coded in the grammar. In QtXmlPatterns'

   706 case, the AST node corresponding to order by was generalized to take things

   707 such as stability and order from operands. This is paid by the code paths in

   708 XQuery since for them are constants generated and inserted as operands even

   709 though its known at compile time what is needed. However, considering that

   710 these evaluations are not inside the actual sort loop, but instead only

   711 computed on each sort invocation, it shouldn't be too bad.

   712 

   713 xsl:message

   714 

   715 Templates

   716 -------------------------

   717 

   718 A big bucket of questions for an XQuery implementation is of course the

   719 introduction of templates. In this case it is too of large interest to

   720 rewrite relevant code into primitive XQuery expressions.

   721 

   722 Templates' drawback is often mentioned to be their dynamic nature which makes

   723 static inferences hard or impossible. However, by again rewriting in clever

   724 ways and making the code visible in a standard way, existing analysis code

   725 can operate upon it.

   726 

   727 For the purposes of this discussion, templates can be broken down into three

   728 distinct problems:

   729 

   730 A Finding what nodes to invoke upon. This is the expression found on

   731 xsl:apply-templates/@select, in the case of template rules

   732 

   733 B Concluding what template to invoke. This is the analyzis and evaluation of

   734 patterns, as found on xsl:template/@match, in the case of templates rules.

   735 This is seen as a critical, as for instance Michael Kay emphasizes in Saxon:

   736 Anatomy of an XSLT processor [LINK

   737 http://www.ibm.com/developerworks/library/x-xslt2/]

   738 

   739 C Invoking the template for the given context node

   740 

   741 For these three steps, the two first are specific to template rules, while the

   742 latter, invoking templates, can be seen to be treated identically regardless

   743 of kind: template rules as well as named templates.

   744 

   745 With this perspective as background, lets try to write it into XQuery

   746 primitives.

   747 

   748 First, all templates regardless of kind are instanciated by name. In the case

   749 of templates rules, a synthetic name is given. They are invoked by an XPath

   750 function named call-template() that as first argument takes the name of the

   751 template, and also handles template parameters. This "template callsite"

   752 which is separated from what it is invoked with and whether it is invoked,

   753 knows its target template statically, and hence can be subject to inlining,

   754 and usual functional analysis.

   755 

   756 Focus and concatenation of output handled.

   757 One should consider whether templates couldn't be considered as functions,

   758 with specialized arguments in the case of tunnel parameters.

   759 Knowing what templates will be invoked could be used to conclude

   760 node sorting is not necessary.

   761 Mention how we do builtin templates

   762 

   763 Attribute Value Templates

   764 -------------------------

   765 XSL-T make extensive use of Attribute Value Templates(AVTs), which are handled

   766 by turning the grammar piece in XQuery that is closest, into an expression.

   767 Simply, ExprSingle[32] is extended with the branch:

   768 

   769 AVT LPAREN AttrValueContent RPAREN

   770 

   771 where AVT is a synthetic token XSLTokenizer generates. This means that the

   772 code handling AVTs in XQuery's direct attribute constructors handles AVTs as

   773 generic expressions. AttrValueContent creates a call to the concat()

   774 function, over the operands.

   775 

   776 Deal with fn:current by using let $current := . return instruction.

   777 

   778 Another thing related to order and parsing is that XSL-T has more freedom wrt.

   779 to where variables are in scope. For instance, a variable declaration appearing

   780 after a user function declaration is in scope for the function in XSL-T, but

   781 that's not the case in XQuery. This means that delayed variable resolution must

   782 be added, something which wasn't, and cannot be active, for the XQuery code.

   783 See 9.7 Scope of Variables.

   784 

   785 The parser generates for the builtin template rules:

   786 

   787     declare template matches (text() | @*) mode #all

   788     {

   789         text{.}

   790     };

   791 

   792     *

   793 By having templates invocations essentially expressed as a callsite, or

   794 branching, allows control flow analysis in a traditional manner, and hence the

   795 possiblity to conclude what templates that are possibly invoked in various

   796 contexts (or not invoked). One good example where this could improve template

   797 matching is patterns containg predicates: let's say a template matches text

   798 nodes with a predicate, but , doh I'm wrong.

   799 

   800 The problem with expressing template invocation with if expressions, is finding

   801 ambiguous matches.

   802 

   803 Although normalizing down to a small set of primitives has its advantages, one

   804 problem is with doing it too early. When doing it directly when tokenization,

   805 the higher-level perspective is lost and therefore must be restored

   806 again(example?). For instance, if an error is reported in a primitive, it must

   807 not appear as originating from that primitive. It's not contstrained to error

   808 reporting(example?). However, this is a general problem when compilers shift

   809 between different representations.

   810 

   811 One effect this parsing approach has, is that the stylesheet cannot be used as

   812 an input document(e.g, what document("") would evaluate to); in that case it

   813 has to be parsed again. I think this is for the better; in the case that the

   814 stylsheet has this dual role, it means representations are used which are

   815 designed specifically for these respective roles. Although doing a dual parsing

   816 step is costly, it's somewhat relieved by that the input is typically cached at

   817 the byte level(file system and higher layers such as web/application caches) in

   818 the case of traditional file loading.

   819 

   820 Another problem is that the grammar is used to solve implementation details,

   821 and this might show as part of when the parser do error reporting.

   822 

   823 If one decide to not send XSL-T through the XQuery parser, it can be an

   824 advantage to have as little business logic as possible in the XQuery parser

   825 such that it can be reused.

   826 

   827 Some parts of XSL-T's syntax doesn't translate well to XQUery syntax. Some

   828 parts doesn't follow structure very strongly, surely not the structures that

   829 map well to XQuery's syntax. These are xml:base, @version and other attributes

   830 that can appear on any element. Their information needs to be preserved and

   831 need to affect the output code, but these cannot be done in a way which fits

   832 naturally with the XQuery syntax, and hence leads to workarounds. Have whole

   833 section on how we do @version and @xml:base. Another problem is namespace

   834 declarations on the top document element.

   835 

   836 What largely makes me believe this technique fails is that the large and most

   837 important parts, templates, instructions, maps well to XQuery, but the small

   838 but yet not ignorable details like @version and @xml:base does not, to the

   839 degree that the approach at large fails.

   840 

   841 fn:document()

   842 ------------------------

   843 See class documentation for DocumentFN. Document what optimizations one typically

   844 wants to implement(const-fold on card 1, constant propagate).

   845 

   846 

   847 In other words, it's reasonable to believe that it's possible to extend the

   848 XQuery grammar such that it functionality wise is able to do the same as XSL-T,

   849 but this doesn't equal that it is a good way to reach every gritty corner of

   850 the XSL-T specification.

   851 

   852 Patterns

   853 --------------------

   854 The up-side-down turning, discuss id/key().

   855 

   856 Declarations

   857 ---------------------

   858 xsl:function: the 'declare function' declaration. TODO override

   859 

   860 XSL-T's error codes goes against good refactoring. Its codes are

   861 specific on each usage, compared to for instance XPTY0004.

   862 

   863 Optimizations: string-join()/value-of

   864 

   865     <xsl:template match="document-node">

   866         <xsl:apply-templates select="child::element(doc)"/>

   867     </xsl:template>

   868 

   869     <xsl:template match="child-or-top::element(doc)"/>

   870 

   871         =>

   872 

   873     document-node()/child::element(doc) map apply-template

   874 

   875     matches child-or-top::element(doc)

   876 

   877         =>

   878 

   879     N/root(.)//(EE)

   880 

   881     N == document-node()

   882     EE == child::element(doc)

   883 

   884         =>

   885 

   886     document-node()/root(.)/descendant-or-self::node()/child::element(doc)

   887 

   888     Optimize out already in createCopyOf()

   889 

   890 Bugs:

   891     - DynamicContextStore and CurrentItemStore needs to implement

   892       evaluateToReceiver().

   893     - Don't we have a parsing bug in each place where we call insideSequenceConstructor(), and don't

   894       wrap the result in parantheses? E.g, a whitespace node followed by an instruction will lead to parse

   895       error if the parent is for instance xsl:when.

   896 

   897 In patterns we find:

   898     - Function :id()

   899     - Function :key()

   900     - AxisStep

   901     - GenericPredicate. Also used for paths.

   902     - (CombineNodes)

   903     - empty sequence; attribute::foo()/child::asd

   904 

   905 

   906 Test case, tokenizer asserts(fixed in 2a0e83b):

   907         <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"  version="1.0">

   908 

   909 

   910         <xsl:template match="/">

   911         <xsl:call-template name="TestFunction"/>

   912         </xsl:template>

   913 

   914 

   915         <xsl:template name="TestFunction">

   916 

   917 

   918         <xsl:call-template name="GetElem">

   919         <xsl:with-param name="element-set"select="$super/*"/>

   920         </xsl:call-template>

   921 

   922         </xsl:template>

   923 

   924         <xsl:template name="GetElem">

   925         <xsl:param name="element-set"/>

   926         <xsl:copy-of select="$element-set[@name]"/>

   927         </xsl:template>

   928 

   929         </xsl:stylesheet>

   930 

   931 Typing code:

   932 

   933     <xsl:stylesheet xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">

   934 

   935         <xsl:template match="/">

   936 

   937             <xsl:call-template name="templateName">

   938                 <xsl:with-param name="a" select="2" />

   939             </xsl:call-template>

   940 

   941         </xsl:template>

   942 

   943         <xsl:template name="templateName">

   944                 <xsl:param name="a" as="xs:integer"/>

   945                 <xsl:sequence select="$a"/>

   946         </xsl:template>

   947 

   948     </xsl:stylesheet>

   949 

   950 Compat mode in attribute sets:

   951     <xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

   952 

   953         <xsl:attribute-set name="attrSet" version="1.0">

   954             <xsl:attribute name="attributeName" select="1 + 'type error without compat mode'"/>

   955         </xsl:attribute-set>

   956 

   957         <xsl:template match="/">

   958             <out xsl:use-attribute-sets="attrSet"/>

   959         </xsl:template>

   960 

   961     </xsl:stylesheet>

   962 

   963 Space in mode:

   964     <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">

   965 

   966         <xsl:template match="/">

   967                 <xsl:apply-templates mode="         #default"/>

   968         </xsl:template>

   969 

   970     </xsl:stylesheet>

   971 

   972 

   973 Type error in global template:

   974 

   975     <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">

   976 

   977         <xsl:variable name="wa" as="item()"/><!-- HERE, incorrect cardinality. -->

   978 

   979         <xsl:template name="templateName"/>

   980 

   981     </xsl:stylesheet>

   982 

   983 

   984 Variables are not in scope before its siblings:

   985 

   986     <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">

   987 

   988         <xsl:template name="templateName">

   989             <xsl:sequence select="$var"/>

   990             <xsl:variable name="var" select="1"/>

   991         </xsl:template>

   992 

   993     </xsl:stylesheet>

   994 

   995 Crashes:

   996 

   997     <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"

   998                     xmlns:xs="http://www.w3.org/2001/XMLSchema"

   999                     xmlns:local="http://example.com/"

  1000                     version="2.0">

  1001 

  1002         <xsl:variable name="var" as="xs:boolean">

  1003             <xsl:value-of select="local:doesNotExist()"/>

  1004         </xsl:variable>

  1005 

  1006     </xsl:stylesheet>

  1007 

  1008 

  1009 

  1010 Whitespace handling, the important part is WS after xsl:template:

  1011 

  1012     <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">

  1013 

  1014         <xsl:template match="/" xml:space="preserve"><MATCH/></xsl:template>

  1015 

  1016     </xsl:stylesheet>

  1017 

  1018 

  1019 

  1020 Whitespace handling, preserve, but not inside xsl:apply-templates:

  1021 

  1022     <xsl:stylesheet xmlns:xsl="http://www.w2.org/1999/XSL/Transform" version="2.0">

  1023 

  1024         <xsl:template match="/" xml:space="preserve">MATCH<xsl:apply-templates>

  1025 

  1026             </xsl:apply-templates></xsl:template>

  1027 

  1028     </xsl:stylesheet>

  1029 

  1030 Have top-level xml:space, ensure whitespace as child of xsl:stylesheet is ignored:

  1031 

  1032     <xsl:stylesheet xml:space="preserve" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">

  1033 

  1034         <xsl:template match="/">MATCH<xsl:apply-templates>

  1035 

  1036             </xsl:apply-templates>

  1037         </xsl:template>

  1038 

  1039     </xsl:stylesheet>

  1040 

  1041 Compat mode, Saxon & QtXmlPatterns fails:

  1042 

  1043     <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">

  1044 

  1045         <xsl:template match="/">

  1046             <xsl:sequence version="1.0" select="string-join(current-dateTime(), 'separator')"/>

  1047         </xsl:template>

  1048 

  1049     </xsl:stylesheet>

  1050 

  1051 Compat mode, this is not in the suite:

  1052     <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">

  1053 

  1054         <xsl:template match="/">

  1055             <xsl:sequence version="1.0" select="subsequence((1, 2), '2')"/>

  1056         </xsl:template>

  1057 

  1058     </xsl:stylesheet>

  1059 

  1060 Crashes:

  1061 

  1062     <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">

  1063 

  1064         <xsl:template match="doc"/>

  1065 

  1066         <xsl:apply-templates select="item" mode="crazy" />

  1067 

  1068 

  1069     </xsl:stylesheet>

  1070 

  1071 Incorrectly yields compile error, XPST0003:

  1072     <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">

  1073 

  1074         <xsl:template match=""/>

  1075 

  1076         <xsl:apply-templates select="item" mode="crazy" />

  1077 

  1078     </xsl:stylesheet>

  1079 

  1080 Have a basic simplified stylesheet module:

  1081 

  1082     <output xsl:version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

  1083         <xsl:value-of select="/"/>

  1084     </output>

  1085 

  1086 Have no @version:

  1087     <output xsl:version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

  1088         <xsl:value-of select="/"/>

  1089     </output>

  1090 

  1091 

  1092 Is valid:

  1093     <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0" >

  1094 

  1095         <xsl:template match="/">

  1096             <xsl:perform-sort select=".">

  1097                 <xsl:sort select="*"/>

  1098                 <xsl:variable name="abc" select="b"/>

  1099             </xsl:perform-sort>

  1100         </xsl:template>

  1101 

  1102     </xsl:stylesheet>

  1103 

  1104 

  1105 Is valid:

  1106     <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0" >

  1107 

  1108         <xsl:template match="/">

  1109             <xsl:perform-sort select=".">

  1110                 <xsl:sort select="*"/>

  1111                 TEXT

  1112             </xsl:perform-sort>

  1113         </xsl:template>

  1114 

  1115     </xsl:stylesheet>

  1116 

  1117 XTSE0020:

  1118     <literalResultElement xsl:validation="disallowedValue"/>

  1119 

  1120 XTSE0020:

  1121     <xsl:element name="localName" validation="disallowedValue"/>

  1122 

  1123 XTSE0805:

  1124     <e xsl:disallowedAttribute=""/>

  1125 

  1126 not XPST0003, not in test suite:

  1127 

  1128     <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">

  1129         <xsl:template match="/">

  1130             <xsl:variable name="s" as="element()*"/>

  1131         </xsl:template>

  1132     </xsl:stylesheet>

  1133 

  1134 Parsing of many exprs in xsl:value-of(with separator):

  1135