Monthly Archives: August 2013

Testing test suite frameworks

Now that I am starting to add more features I was also beginning to feel the need to test the framework itself, but how do you test test suite frameworks?

I did so by adding a new layer on top of the framework: this layer runs test suites and check that their results are what is expected.

And because the vocabulary  to describe the XForms Unit suites is very experimental and likely to change a lot in the near feature I have kept the format used by this high level layer minimal and distinct from the description of the suites.

In fact the description of these tests is limited to a reference to the test suite to run and a list of expected results:

<test>
    <title>Sample suite</title>
    <suite href="../../suites/hello-world/suite.xml"/>
    <expected>
        <case>
            <assert expected="true"/>
        </case>
    </expected>
</test>

Extract from the tests for m:assertEqual assertions

These tests can then be run. The output format is currently pretty raw:

Screenshot of test results

Despite its crudeness, this tool is really powerful and very helpful for changes such as the update in the suite vocabulary committed a few minutes ago.

My next steps is to add assertions on other control properties (read/write status, validity and optionality).

Enabling test automation

Most of the usefulness of unit test environment comes from their ability to automate the tests.

With the principles used in this project, automation can be achieved by submitting the tests results once they become available. Translated into XForms terms, this means submitting the results in an action at the end of test  which are themselves triggered by an xforms-ready event .

For server side implementations, using a submission with @replace=”all” makes a lot of sense because the tests can be performed server side without any browser interaction and directly return test results.

The basic feature to enable this is thus the ability to perform submissions with @replace=”all” on xforms-ready events.

This simple repro case checks this ability:

  <html xmlns="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms"
    xmlns:ev="http://www.w3.org/2001/xml-events" >
    <head>
        <title>Submission on xforms-ready</title>
        <xf:model id="model">
            <xf:instance id="instance">
                <data xmlns="">Whatever</data>
            </xf:instance>

            <xf:submission id="echo" ref="instance('instance')" method="POST"
                resource="http://localhost:8088/exist/apps/betterform/sandbox/echo.xq" replace="all"/>

            <xf:send ev:event="xforms-ready" submission="echo"/>

        </xf:model>
    </head>
    <body> </body>
</html>

xforms-ready-submission.xhtml

Unfortunately this does work with Orbeon Forms but not with betterFORM nor with XSLTForms.

Trying to develop inter-operable XForms applications happens to be very similar to trying to develop inter-operable web applications but just harder: there are no JavaScript libraries to hide incompatibilities, nobody seems to care much and the differences are not documented anywhere!

I have created a page to document XForms interoperability issues, hoping that this work may be useful for others.

To come back to test automation, Orbeon Forms has defined an extension to perform local echo submissions without client/server interactions. With this extension you don’t have to rely on a web service to replace the page by its test instance and that facilitates the development of automated tests running in pipelines.

Of course this won’t be possible with a client side implementation such as XSLTForms and for these implementations we will probably have to run a browser on a virtual screen such as Xvfb.

A vocabulary to describe test suites

That leads on to the question, at what level should we test?

Stephen Cameron, 2013-08-22 06:54

Indeed and we have to remember that XForms is following MVC principles.

In an XForms form the model is embedded in an xf:model element, the view is elsewhere in document’s body and the controller is spread out within the entire document using separate elements names (so called actions) and the risk of confusion between these three components is limited.

In test cases on the contrary the pattern is to invoke actions from the controller to simulate real world situations and to perform checks within either the view or the controller and I think that the risk of confusion is much higher and that it will be useful to clearly separate these levels.

This separation cannot easily be done adding containers for the model, view or controller since a test suite will typically involved a number of individual actions and tests on each of these layers and I think that this is a very good use case for namespaces.

My proposal for the vocabulary that will describe the test suites is thus to use four different namespaces:

  • http://xformsunit.org/namespaces/suite/ for the suite itself
  • http://xformsunit.org/namespaces/model/ for the model
  • http://xformsunit.org/namespaces/view/ for the view
  • http://xformsunit.org/namespaces/controller/ for the controller

The simple suite described in the blog post which has introduced the project would then become:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-model href="../../suite.rng" type="application/xml" schematypens="http://relaxng.org/ns/structure/1.0"?>
<?xml-model href="../../suite.rng" type="application/xml" schematypens="http://purl.oclc.org/dsdl/schematron"?>
<suite xmlns:xh="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="http://xformsunit.org/namespaces/suite/"
    xmlns:m="http://xformsunit.org/namespaces/model/" xmlns:v="http://xformsunit.org/namespaces/view/"
    xmlns:c="http://xformsunit.org/namespaces/controller/">
    <form src="hello-world.xhtml"/>

    <!-- The test cases -->
    <case id="test-greetings">
        <title>Test that greetings are correctly set</title>
        <c:setvalue ref="instance('instance')/PersonGivenName">Eric</c:setvalue>
        <m:assertEqual>
            <m:actual ref="instance('instance')/Greetings"/>
            <m:expected>Hello Eric. We hope you like XForms!</m:expected>
            <m:message>The greetings should be the concatenation of "Hello ", the given name and ". We hope you like
                XForms!".</m:message>
        </m:assertEqual>
    </case>
</suite>

suite.xml

This one uses the controller to set a value and performs a test on the model level. And yes, you’ve see the <?xml-model?> right, there is a schema for this markup. The normative schema is written in Examplotron:

<?xml version="1.0" encoding="UTF-8"?>
<suite xmlns="http://xformsunit.org/namespaces/suite/" xmlns:m="http://xformsunit.org/namespaces/model/"
    xmlns:v="http://xformsunit.org/namespaces/view/" xmlns:c="http://xformsunit.org/namespaces/controller/"
    xmlns:xh="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:eg="http://examplotron.org/0/">
    <!-- Reference to the form to test -->
    <form src="hello-world.xhtml"/>

    <!-- The test cases -->
    <case id="test-greetings" eg:occurs="*" eg:content="eg:interleave">
        <!-- Title -->
        <title>Test that the greetings are displayed and enabled.</title>
        <!-- Set a value (similar to XForms' setvalue action) -->
        <c:setvalue ref="instance('instance')/PersonGivenName" eg:occurs="*">Eric</c:setvalue>
        <!-- Check that an instance node is equal to its expected value -->
        <m:assertEqual eg:occurs="*">
            <!-- Actual value -->
            <m:actual ref="instance('instance')/Greetings"/>
            <!-- Expected -->
            <m:expected>Hello Eric. We hope you like XForms!</m:expected>
            <!-- Message to display when the test fails -->
            <m:message>The greetings should be the concatenation of "Hello ", the given name and ". We hope you like
                XForms!".</m:message>
        </m:assertEqual>
        <!-- Check if a control is enabled -->
        <v:assertEnabled eg:occurs="*">
            <!-- Identification of the control. Being an element leaves more flexibility to use other means than id/idref -->
            <v:control idref="greetings-control"/>
            <!-- Expected value (boolean) -->
            <v:expected>true</v:expected>
            <!-- Message to display if the test fails -->
            <v:message>The greetings control should be enabled at that point.</v:message>
        </v:assertEnabled>
    </case>
</suite>

suite.eg

Examplotron schemas are also document samples and this one performs an additional test on the view to check the status of a control.

This schema can be compiled into RELAX NG using either the XML or the compact syntax.

And now I need to implement the <v:assertEnabled/> which, given my recent interoperability misadventures, won’t be that easy for betterFORM and XSLTForms!

As always, your feedback is most welcome and needed.

A simple form reveals four issues

After an idea that came up in the comments on my last post, I have written a simple form to test how xforms-enabled/xforms-disabled events could be tracked to determine the status of controls.

The form did work as expected with Orbeon Forms but doesn’t happen to work with neither betterFORM not XSLTForms and I have sent reports accordingly on  the Betterform-users and xsltforms-support mailing lists.

These issues are really blocking to pursue this idea which would allow to test control’s statuses. I don’t see what can be possibly wrong in the code that I have written and do hope that we’ll find a workaround or get fixes soon.

While doing these tests I have also noticed that betterFORM was not reacting like Orbeon Forms and XSLTForms when changing the selection in xf:select controls: with betterFORM the value is not updated in the instance until the focus leaves the control while it is immediately updated for Orebon Forms and XSLTForms.

Developing mostly for Orbeon Forms this behavior seemed natural to me but the W3C recommendation clearly states that betterFORM is right when there is no @incremental=”true” attribute. Here again I have reported the issues on Xsltforms-support and orbeon-forms.

This second issue is minor and doesn’t really impact XForms Unit but this clearly show what happens when you try to develop forms that should be inter-operable in the three main open source XForms implementation: even simple forms are a nightmare like here where I had to fill four reports!