Tag Archives: automation

Testing test suite frameworks

Now that I am starting to add more features I was also beginning to feel the need to test the framework itself, but how do you test test suite frameworks?

I did so by adding a new layer on top of the framework: this layer runs test suites and check that their results are what is expected.

And because the vocabulary  to describe the XForms Unit suites is very experimental and likely to change a lot in the near feature I have kept the format used by this high level layer minimal and distinct from the description of the suites.

In fact the description of these tests is limited to a reference to the test suite to run and a list of expected results:

<test>
    <title>Sample suite</title>
    <suite href="../../suites/hello-world/suite.xml"/>
    <expected>
        <case>
            <assert expected="true"/>
        </case>
    </expected>
</test>

Extract from the tests for m:assertEqual assertions

These tests can then be run. The output format is currently pretty raw:

Screenshot of test results

Despite its crudeness, this tool is really powerful and very helpful for changes such as the update in the suite vocabulary committed a few minutes ago.

My next steps is to add assertions on other control properties (read/write status, validity and optionality).

Enabling test automation

Most of the usefulness of unit test environment comes from their ability to automate the tests.

With the principles used in this project, automation can be achieved by submitting the tests results once they become available. Translated into XForms terms, this means submitting the results in an action at the end of test  which are themselves triggered by an xforms-ready event .

For server side implementations, using a submission with @replace=”all” makes a lot of sense because the tests can be performed server side without any browser interaction and directly return test results.

The basic feature to enable this is thus the ability to perform submissions with @replace=”all” on xforms-ready events.

This simple repro case checks this ability:

  <html xmlns="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms"
    xmlns:ev="http://www.w3.org/2001/xml-events" >
    <head>
        <title>Submission on xforms-ready</title>
        <xf:model id="model">
            <xf:instance id="instance">
                <data xmlns="">Whatever</data>
            </xf:instance>

            <xf:submission id="echo" ref="instance('instance')" method="POST"
                resource="http://localhost:8088/exist/apps/betterform/sandbox/echo.xq" replace="all"/>

            <xf:send ev:event="xforms-ready" submission="echo"/>

        </xf:model>
    </head>
    <body> </body>
</html>

xforms-ready-submission.xhtml

Unfortunately this does work with Orbeon Forms but not with betterFORM nor with XSLTForms.

Trying to develop inter-operable XForms applications happens to be very similar to trying to develop inter-operable web applications but just harder: there are no JavaScript libraries to hide incompatibilities, nobody seems to care much and the differences are not documented anywhere!

I have created a page to document XForms interoperability issues, hoping that this work may be useful for others.

To come back to test automation, Orbeon Forms has defined an extension to perform local echo submissions without client/server interactions. With this extension you don’t have to rely on a web service to replace the page by its test instance and that facilitates the development of automated tests running in pipelines.

Of course this won’t be possible with a client side implementation such as XSLTForms and for these implementations we will probably have to run a browser on a virtual screen such as Xvfb.