Squash TF Java Junit Runner¶

Runner Functions¶

The runner is transparent and should be able to run any Maven Junit 4 or 5 project without any need to modify it. Indeed the Mojos are implemented in such a way that they are fully autonomous and do not need any project specific configuration to run.

List implemented Junit tests¶

This Mojo enables one to list as a Json file the available implemented tests. In order to do so one only needs to run the following command :

mvn clean compile test-compile org.squashtest.ta.galaxia:squash-tf-junit-runner-maven-plugin:1.0.0-RELEASE:list

The command is structured as follows

mvn : launch Maven
clean : (Optional) One of Maven default goals that enables one to clean everything that has been previously build by Maven.
compile : One of Maven default goals that enables one to compile code that is stored in src/main.
test-compile : One of Maven default goals that enables one to compile code that is stored in src/test.
org.squashtest.ta.galaxia:squash-tf-junit-runner-maven-plugin:1.0.0-RELEASE:list : the actual listing Mojo provided by the Squash TF Java Junit Runner. It lists all Junit tests that can be discovered (in the Junit sense) in the code compiled during the “compile” and “test-compile” phases.

The result should be a Json file named testTree.json located at target/squashTA/test-tree. It is a simple JavaScript table listing available test grouped by “ecosystems”. Ecosystem naming follows the convention explained in the Junit tests reference scheme section above.

‘list’ goal with Metadata¶

If there are Metadata in the current test project, the goal “list” searches and checks if all metadata in this JUnit project respect the conventions for writing and using Squash TF metadata. (See Metadata in JUnit runner for more information about Metadata syntax conventions)

The goal will check through the project, collect all the metadata error(s) if any and lead to the FAILURE. Otherwise, a SUCCESS result will be obtained.

Metadata error(s), if found, will be grouped by test names.

Listing test JSON report with Metadata¶

If the build is successful, the generated report (JSON file) will contain the metadata associated with each of the test scripts.

{
        "timestamp": "2019-09-23T14:56:24.761+0000",
        "name": "tests",
        "contents": [{
                    "name": "maven.test.bundle:com.example.project.CalculatorTest2",
                    "contents": [{
                        "name": "testAdd()",
            "metadata": {},
            "contents": null
        }, {
            "name": "This is my test",
            "metadata": {
                "key1..": null
            },
            "contents": null
        }, {
            "name": "testAddWithTFMetadata2()",
            "metadata": {
                "key2": ["value2"]
            },
            "contents": null
        }, {
            "name": "testAddWithTFMetadata3()",
            "metadata": {
                "Key3-": null,
                "key_4": ["value4"],
                "Key53": ["value4-","value4-"]
            },
            "contents": null
        }
    ]
}

Note

To ignore thoroughly Metadata during the listing process as well as in the report, insert tf.disableMetadata property after the goal “list”.

mvn clean compile test-compile org.squashtest.ta.galaxia:squash-tf-junit-runner-maven-plugin:1.0.0-RELEASE:list -Dtf.disableMetadata=true

Junit test Running¶

This Mojo enables one to run a selection of, or all possible, Junit tests and report their execution. In order to do so one only needs to run the following command :

mvn clean compile test-compile org.squashtest.ta.galaxia:squash-tf-junit-runner-maven-plugin:1.0.0-RELEASE:run

mvn : launch Maven
clean : (Optional) One of Maven default goals that enables one to clean everything that has been previously build by Maven.
compile : One of Maven default goals that enables one to compile code that is stored in src/main.
test-compile : One of Maven default goals that enables one to compile code that is stored in src/test.
org.squashtest.ta.galaxia:squash-tf-junit-runner-maven-plugin:1.0.0-RELEASE:run : the actual running Mojo provided by Squash TF Java Junit Runner. It runs the Junit tests that can be discovered (in the Junit sense) in the code compiled during the “compile” and “test-compile” phases.

By default the whole collection of tests available in the project will be executed. A summary of the execution is reported and available at target/squashTA/html-reports/squash-ta-report.html . A more detailed version of the report providing context in the case of technical error is also produced and available at target/squashTA/html-details/squash-ta-report.html. Finally a surfire report is also produced and available at target/squashTA/surefire-reports/ .

If one wants to only run a subset of possible test one can provide a list of tests via the Maven property “tf.test.suite”. Two mechanism are possible:

Mimic TM-TF link and provide a list of selected test via a Json file. In this scenario the tf.test.suite parameter should be given the value “{file:testsuite.json}” and the testsuite.json should be put right to the project pom.
Provide a CSV like line, where qualified tests names are listed separated by semicolons

In both cases test convention should follow the one used by the listing Mojo and described in the Junit tests reference scheme section above.

Attention

If there are syntax-error-metadata in the running test script(s), warning message(s) will be displayed in the console. (See Metadata in JUnit runner for more information about Metadata syntax conventions)

Junit test METADATA Checking¶

As the goal “list”, the goal check-metadata searches and checks if all metadata in this JUnit project respect the conventions for writing and using Squash TF metadata. (See Metadata in JUnit runner for more information about Metadata syntax conventions)

mvn clean compile test-compile org.squashtest.ta.galaxia:squash-tf-junit-runner-maven-plugin:1.1.0-RELEASE:check-metadata

The goal will check through the project, collect all the metadata error(s) if any and lead to the FAILURE. Otherwise, a SUCCESS result will be obtained. (However, no JSON report will be created with a successful check-metadata goal.)

Metadata error(s), if found, will be grouped by test names.

When a JUnit project has duplicate values in a multi-value key on a given test, the check-metadata goal will create a WANING message in the console.

‘check-metadata’ goal with Unicity checking¶

In addition to the normal syntax checking, you can insert the tf.metadata.check property after the goal “check-metadata” to check the unicity of each Metadata Key - Value pair.

mvn clean compile test-compile org.squashtest.ta.galaxia:squash-tf-junit-runner-maven-plugin:1.1.0-RELEASE:check-metadata -Dtf.metadata.check=[valueUnicity]

If there are Metadata Key - Value duplicate(s) existed in the SKF project (even if the syntax is OK), a FAILURE result will be obtained.

Check-metadata with Unicity checking build FAILURE

‘check-metadata’ goal with Unicity checking for specific Keys¶

You can even check the unicity of each Metadata Key - Value pair with just some specific Keys by inserting the second property tf.metadata.check.key after the first one mentioned above.

mvn clean compile test-compile org.squashtest.ta.galaxia:squash-tf-junit-runner-maven-plugin:1.1.0-RELEASE:check-metadata -Dtf.metadata.check=[valueUnicity] -Dtf.metadata.check.keys=[xxx,yyy,zzz]

Important

In the bracket, the key list MUST be a string of characters composed by concatenation from 1 to n keys separated by commas: -Dtf.metadata.check.keys=[xxx,yyy,zzz]

If the list is surrounded by double quotes, spaces are allowed: -Dtf.metadata.check.keys=”[xxx, yyy, zzz]”

It is NOT allowed to have two commas without any key OR only spaces/tabulations between them. (ex: -Dtf.metadata.check.keys=”[xxx, ,yyy,,zzz]”)

Key list is NOT allowed to be either uninitiated or empty. (ex: -Dtf.metadata.check.keys= OR -Dtf.metadata.check.keys=[])

For each searched Metadata key, if there are Key - Value duplicate(s) existed in the SKF project, a FAILURE result will be obtained.

Check-metadata with Unicity checking for specific Keys build FAILURE

Note

If searched Metadata key(s) are not found in any Test files, a WANING message will be raised in the console.

Junit tests reference scheme¶

Tests naming scheme¶

Junit tests detected in the project have to receive unique - and as far as possible readable - names in the runner system. Such names are used in the following contexts :

Listing available tests

Requiring the runner to run a subset of tests

Linking Squash TM test cases to the junit test covering it

Displaying tests results in execution reports

In this context, we define the qualified name of the Junit test as follows :

bundle-name:qualified_class_name/Display Name, where:

bundle-name : name of the code bundle in which the tests are defined. The runner defines two such bundles :

maven.main.bundle : junit code from maven main sources (by default located in the src/main/java and src/main/resources subdirectories)

maven.test.bundle : junit code from maven test sources (by default located in the src/main/java and src/main/resources subdirectories)

qualified_class_name : the dot-separated qualified name of the class where the test is defined, as specified by java conventions.

Display Name : the display name of the test, as defined by junit. By default this is the name of the method that defines the test, but junit allows you to override this using the @DisplayName annotation.

This test name appears in test lists for the TM-TF link, they may also be used to select tests to run from the command line (see below).

There is a small difference in test execution reports : tests are grouped in test groups named ecosystems. These groups are defined like the prefix we add to the Display Name to make qualified test names : according to the code bundle and test class. Ecosystems are named accordingly, but the separator between the bundle and class parts is a dot instead of a colon :

bundle-name.qualified_class_name

In these reports, tests appear under a qualified test result name where the colon and slash separators are also replaced by a dot :

bundle-name.qualified_class_name.Display Name

Tests with the same `@displayName`¶

The use of the @displayName junit annotation allows you to give two tests from the same class the same unqualified display name. And as these tests come from the same class, in the same bundle, this will given them the same qualified display name through the runner.

We consider this bad practice and strongly advise against it, even if you don’t use our runner. Giving two tests the same display name will tend to make reports harder to read: if two or more tests cover exactly the same case, then why use time and work to write more than one test, and read more than one result ? On the other hand, if they test different things you should make yourself a favor and name them differently, to avoid confusion.

If, however, you have inherited such tests, or you have your own reason to go against this advice, the runner will work with this tests in the following way :

All tests with the same qualified display name will be seen as a single test, and listed and executed as such.

All junit tests will be executed and checked as a global unit which result will be the worst result of any test.

You will be able to link this global functional test defined by its common qualified display name to Squash TM test cases as any other automated test, execute the test case and see the result from Squash TM

Known limitation : if several tests share the same qualified display name, the runner will not handle metadata declaration (defined through the use of Squash TF’s @Metadata annotations on the methods) properly. We intend to fix this in a later version.

Metadata in JUnit runner¶

In your JUnit project, you can insert Squash Metadata into test methods via an Annotation named TFMetadata. For example:

@Test
@DisplayName("This is my test with Squash Metadata")
@TFMetadata(key ="key", value ={"value"})
public void testAddWithTFMetadata1() {
    Calculator calculator = new Calculator();
    assertEquals(2, calculator.add(1, 1), "1 + 1 should equal 2");
}

Configuration : new dependency¶

In order to be able to use TF metadata in your Junit project, you have to add a new dependency to your project.

To do so :

Add to the dependencies section of your project pom.xml the following dependency :

<dependency>
  <groupId>org.squashtest.ta.galaxia</groupId>
  <artifactId>squash-tf-galaxia-annotations</artifactId>
  <version>1.0.0-RELEASE</version>
</dependency>

Add to the repositories section (or create it if doesn’t exist) of your project pom.xml the following repository. (This is needed because our dependency is not available on maven central but only in our repository)

<repositories>
  <repository>
    <id>org.squashtest.ta.release</id>
    <name>squashtest test automation - releases</name>
    <url>http://repo.squashtest.org/maven2/releases</url>
  </repository>
</repositories>

Metadata syntax conventions¶

In a Metadata annotation, the key is mandatory. A metadata key MUST be ONE WORD which contains only alphanumeric characters, dashes, underscores and dots. Spaces/tabulation are allowed before and after the word.

Moreover, metadata key is case insensitive and must be unique in a test file, even a test method.

@Test
@DisplayName("For a typical Metadata Key")
@TFMetadata(key ="   Key.01-is_Insensitive-and_MUST-be_UNIQUE      ", value ={"value"})
public void testAddWithTFMetadata2() {
    Calculator calculator = new Calculator();
    assertEquals(3, calculator.add(1, 2), "1 + 2 should equal 3");
}

On the other hand, the value is optional. However, if existed, a metadata value MUST be ONE WORD which contains only alphanumeric characters, dashes, slashes, underscores and dots. Spaces/tabulation are also allowed before and after the word.

Metadata value is case sensitive and must be assigned to a metadata key. It is also possible to have many metadata values associated to a same key.

@Test
@TFMetadata(key ="key1")
public void testAddWithTFMetadata3() {
    Calculator calculator = new Calculator();
    assertEquals(4, calculator.add(1, 3), "1 + 3 should equal 4");
}

@Test
@TFMetadata(key ="key2", value ={})
public void testAddWithTFMetadata4() {
    Calculator calculator = new Calculator();
    assertEquals(5, calculator.add(1, 4), "1 + 4 should equal 5");
}

@Test
@TFMetadata(key ="key3", value ={"   pathTo/Value.03-is_Sensitive      "})
public void testAddWithTFMetadata5() {
    Calculator calculator = new Calculator();
    assertEquals(6, calculator.add(1, 5), "1 + 5 should equal 6");
}

@Test
@TFMetadata(key ="key4", value ={"value1", "Value2", "value3"})
public void testAddWithTFMetadata6() {
    Calculator calculator = new Calculator();
    assertEquals(7, calculator.add(1, 6), "1 + 6 should equal 7");
}

A test method can be associated with from zero to many Metadata.

@Test
@DisplayName("No metadata")
public void testAddWithTFMetadata7() {
    Calculator calculator = new Calculator();
    assertEquals(8, calculator.add(1, 7), "1 + 7 should equal 8");
}

@Test
@DisplayName("With one metadata")
@TFMetadata(key ="key", value ={"value"})
public void testAddWithTFMetadata8() {
    Calculator calculator = new Calculator();
    assertEquals(9, calculator.add(1, 8), "1 + 8 should equal 9");
}

@Test
@DisplayName("With many metadata")
@TFMetadata(key ="key1", value ={})
@TFMetadata(key ="key2", value ={"value"})
@TFMetadata(key ="key3", value ={"value1", "value2"})
public void testAddWithTFMetadata9() {
    Calculator calculator = new Calculator();
    assertEquals(10, calculator.add(1, 9), "1 + 9 should equal 10");
}

Important

Please ensure that all Metadata keys in every JUnit method of a Test script are unique.

Use metadata for TM - TF autolink¶

TF metadata handles the TM - TF autolink. (TM - TF autolink is available since TM 1.20.0 and Java Junit Runner 1.1.0) Autolink is a feature to ease the link between a TM test case and a test automation script. On TM side, a UUID is now provided (when the workflow is activated) :

This UUID is used as an identifier.

In your automation test add a TF Metadata which key is linked-TC and value is the UUID from the corresponding TM test case. As you can see in the example below, it’s possible to link many TM test case to the same automation test (two UUID are set in the “value”):

Creating projects¶

Starting a new project¶

In case you are starting a new Maven Java Junit project from scratch, a Maven archetype is provided to help you generate a correctly structured project with recommended dependencies. The archetetype group Id is org.squashtest.ta.galaxia, the archetype artifact Id is squash-tf-junit-runner-project-achetype and the current version is 1.0.0-RELEASE. Hence, to use it simply run the following command :

mvn archetype:generate -DarchetypeGroupId=org.squashtest.ta.galaxia -DarchetypeArtifactId=squash-tf-junit-runner-project-archetype -DarchetypeVersion=1.0.0-RELEASE

Tests implementation: sample with SoapUI Smartbear Api¶

Once your project is created, you just have to implement your tests as Junit tests according to the version of Junit chosen.

Here is an example to run a REST test with the open source API proposed by SoapUI Smartbear.

We used:

a REST test given in the tutorial installed with version 5.5.0 of SoapUI Smartbear software.
the open source SoapUI Smartbear API version 5.1.3
the TF JUnit runner version 1.0.0-RELEASE
junit (jupiter) version 5.2.0

We have set in the POM’s project the version of junit chosen:

You need to add the external dependency the SoapUI API in your POM file as follows:

In this example, you must also add in your POM a block for the Smartbear repository:

Create your test class and implement it. Your project now looks like this:

Launch the TF junit runner:

The HTML results file of the execution squash-ta-report.html is available in the subdirectory squashTA/html-report of the target folder.

Note: If you have not started the server mock, as indicated in the SoapUI tutorial, your test will fail on a connection error.

Overview¶

The Squash Test Factory (Squash TF) Java Junit Runner aims to provide a seamless integration to our ecosystem when automated test are already (will be) implemented using Java as a language and Junit as the underlying test framework.

As a Squash TF runner its main goal is twofold :

List in JavaScript Object Notation (Json) format the available implemented tests
Run a selection (that can include all available tests) and report the execution. In the case where the execution order originates from Squash Test Management (Squash TM), test status and report are sent back to Squash TM.

The actual implementation of the Runner is based on the Maven technology. More precisely each major feature (“List” and “Run”) is provided via an implementation of a Maven Mojo.