Tag Archives: markup

Testing test suite frameworks

Now that I am starting to add more features I was also beginning to feel the need to test the framework itself, but how do you test test suite frameworks?

I did so by adding a new layer on top of the framework: this layer runs test suites and check that their results are what is expected.

And because the vocabulary  to describe the XForms Unit suites is very experimental and likely to change a lot in the near feature I have kept the format used by this high level layer minimal and distinct from the description of the suites.

In fact the description of these tests is limited to a reference to the test suite to run and a list of expected results:

<test>
    <title>Sample suite</title>
    <suite href="../../suites/hello-world/suite.xml"/>
    <expected>
        <case>
            <assert expected="true"/>
        </case>
    </expected>
</test>

Extract from the tests for m:assertEqual assertions

These tests can then be run. The output format is currently pretty raw:

Screenshot of test results

Despite its crudeness, this tool is really powerful and very helpful for changes such as the update in the suite vocabulary committed a few minutes ago.

My next steps is to add assertions on other control properties (read/write status, validity and optionality).

A vocabulary to describe test suites

That leads on to the question, at what level should we test?

Stephen Cameron, 2013-08-22 06:54

Indeed and we have to remember that XForms is following MVC principles.

In an XForms form the model is embedded in an xf:model element, the view is elsewhere in document’s body and the controller is spread out within the entire document using separate elements names (so called actions) and the risk of confusion between these three components is limited.

In test cases on the contrary the pattern is to invoke actions from the controller to simulate real world situations and to perform checks within either the view or the controller and I think that the risk of confusion is much higher and that it will be useful to clearly separate these levels.

This separation cannot easily be done adding containers for the model, view or controller since a test suite will typically involved a number of individual actions and tests on each of these layers and I think that this is a very good use case for namespaces.

My proposal for the vocabulary that will describe the test suites is thus to use four different namespaces:

  • http://xformsunit.org/namespaces/suite/ for the suite itself
  • http://xformsunit.org/namespaces/model/ for the model
  • http://xformsunit.org/namespaces/view/ for the view
  • http://xformsunit.org/namespaces/controller/ for the controller

The simple suite described in the blog post which has introduced the project would then become:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-model href="../../suite.rng" type="application/xml" schematypens="http://relaxng.org/ns/structure/1.0"?>
<?xml-model href="../../suite.rng" type="application/xml" schematypens="http://purl.oclc.org/dsdl/schematron"?>
<suite xmlns:xh="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="http://xformsunit.org/namespaces/suite/"
    xmlns:m="http://xformsunit.org/namespaces/model/" xmlns:v="http://xformsunit.org/namespaces/view/"
    xmlns:c="http://xformsunit.org/namespaces/controller/">
    <form src="hello-world.xhtml"/>

    <!-- The test cases -->
    <case id="test-greetings">
        <title>Test that greetings are correctly set</title>
        <c:setvalue ref="instance('instance')/PersonGivenName">Eric</c:setvalue>
        <m:assertEqual>
            <m:actual ref="instance('instance')/Greetings"/>
            <m:expected>Hello Eric. We hope you like XForms!</m:expected>
            <m:message>The greetings should be the concatenation of "Hello ", the given name and ". We hope you like
                XForms!".</m:message>
        </m:assertEqual>
    </case>
</suite>

suite.xml

This one uses the controller to set a value and performs a test on the model level. And yes, you’ve see the <?xml-model?> right, there is a schema for this markup. The normative schema is written in Examplotron:

<?xml version="1.0" encoding="UTF-8"?>
<suite xmlns="http://xformsunit.org/namespaces/suite/" xmlns:m="http://xformsunit.org/namespaces/model/"
    xmlns:v="http://xformsunit.org/namespaces/view/" xmlns:c="http://xformsunit.org/namespaces/controller/"
    xmlns:xh="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:eg="http://examplotron.org/0/">
    <!-- Reference to the form to test -->
    <form src="hello-world.xhtml"/>

    <!-- The test cases -->
    <case id="test-greetings" eg:occurs="*" eg:content="eg:interleave">
        <!-- Title -->
        <title>Test that the greetings are displayed and enabled.</title>
        <!-- Set a value (similar to XForms' setvalue action) -->
        <c:setvalue ref="instance('instance')/PersonGivenName" eg:occurs="*">Eric</c:setvalue>
        <!-- Check that an instance node is equal to its expected value -->
        <m:assertEqual eg:occurs="*">
            <!-- Actual value -->
            <m:actual ref="instance('instance')/Greetings"/>
            <!-- Expected -->
            <m:expected>Hello Eric. We hope you like XForms!</m:expected>
            <!-- Message to display when the test fails -->
            <m:message>The greetings should be the concatenation of "Hello ", the given name and ". We hope you like
                XForms!".</m:message>
        </m:assertEqual>
        <!-- Check if a control is enabled -->
        <v:assertEnabled eg:occurs="*">
            <!-- Identification of the control. Being an element leaves more flexibility to use other means than id/idref -->
            <v:control idref="greetings-control"/>
            <!-- Expected value (boolean) -->
            <v:expected>true</v:expected>
            <!-- Message to display if the test fails -->
            <v:message>The greetings control should be enabled at that point.</v:message>
        </v:assertEnabled>
    </case>
</suite>

suite.eg

Examplotron schemas are also document samples and this one performs an additional test on the view to check the status of a control.

This schema can be compiled into RELAX NG using either the XML or the compact syntax.

And now I need to implement the <v:assertEnabled/> which, given my recent interoperability misadventures, won’t be that easy for betterFORM and XSLTForms!

As always, your feedback is most welcome and needed.