added testng.xml and modified pom for running tests locally

[jamesknowsbest]

Issue

When uploading appium testNG test package to aws Device Farm it is required to have the testng.xml file in the root of the jar. This is not clearly documented and this pull request should help with resolving and preventing future issues customers may face

Short Description

This pull request describes how to include the testng.xml files in the root of the jar file created using a maven command from awslabs github project.
Resolution

What I did

To exemplify using the testng.xml file we'll use the sample appium java project [1]and the sample android project [2]from awslabs github page.

• After downloading or cloning the appium java testng sample project, and extracting it if necessary, change the active directory to that project.

cd aws-device-farm-appium-tests-for-sample-app/

• Then we'll create the /src/test/resources directory to take advantage of the super pom default configuration:

mkdir src/test/resources

Content added to the src/test/resources directory gets added because of this tag in the super pom which is configured to added all files to the jar that are in the resources directory [3] . To view the super pom of the project run this command:

mvn help:effective-pom

Alternatively, we can implement the testResources tags in the pom.xml to explictly reference another directory besides src/test/resources as in this maven doc:

https://maven.apache.org/pom.html#The_Super_POM

From docs:

<testResources>
      <testResource>
        <directory>${project.basedir}/src/test/resources</directory>
      </testResource>
    </testResources>

I then created the testng.xml file in the src/test/resources directory and added this content to it to only run the AlertPageTest class:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Default Suite">
    <test name="test">
        <classes>
            <class name="Tests.AlertPageTest"/> <!--Package.ClassName-->
        </classes>
    </test>
</suite>

• Then I modified the pom.xml to include the surefire plugin to reference this testng.xml file which is necessary for running the tests locally as testng tests but optional if the project is just for Device Farm [4] :

<plugin> 
    <groupId>org.apache.maven.plugins</groupId> 
    <artifactId>maven-surefire-plugin</artifactId> 
    <version>2.12.4</version> 
    <configuration>
       <suiteXmlFiles> <suiteXmlFile>src/test/resources/testng.xml</suiteXmlFile></suiteXmlFiles>
    </configuration>
</plugin>

• Once this is done I then deleted the contents of the target directory and packaged the tests using this maven command:

mvn clean package -DskipTests=true

This created the zip-with-dependencies.zip file in the target directory with the two jar files. I extracted the jar to verify that the test.xml was in the root (see screenshot targetDirectoryBeforeExtraction.png and targetDirectoryAfterExtraction.png) using this command [5]:

jar xf nameOfTheProjectFromPom-1.0-SNAPSHOT-tests.jar
Related information:

[1] https://github.com/awslabs/aws-device-farm-appium-tests-for-sample-app
[2] https://github.com/awslabs/aws-device-farm-sample-app-for-android
[3]https://maven.apache.org/pom.html#Resources
[4]https://maven.apache.org/surefire/maven-surefire-plugin/examples/testng.html
[5]http://docs.aws.amazon.com/devicefarm/latest/developerguide/troubleshooting-appium-web-java-testng.html

[sapins]

How can we use same thing on AWS device farm?

[jamesknowsbest]

So if you add the testng.xml to the root of the jar it will be valid in Device Farm. The above offers some options for doing that. However these annotations cannot be used currently inside of the test classes:

BeforeSuite,
AfterSuite,
BeforeClass,
AfterClass,
BeforeGroups,
AfterGroups,
BeforeTest,
AfterTest,
DataProvider,
Factory

To use them you will need to make a base class and then extend the base class using the test classes.

[Edit]
You should now be able to put these annotations anywhere in the test package.

[kevin-liu-agriwebb]

@jamesknowsbest
Currently even the Test annotation is note supported in device farm, nor BeforeTest etc. See below error from parsed result of device farm.

When will TestNG annotations be supported in device farm?

`Failed to generate the test suites from the test package. See the information below for more details.

Jul 20, 2018 1:41:32 AM com.amazon.aatp.DryRunAnnotationTransformer transform
INFO: Found @test annotation on method: public void com.agriwebb.notebook2.test.LoginAndSyncTest.login()
[TestNG] [ERROR]
Cannot instantiate class com.agriwebb.notebook2.test.LoginAndSyncTest`

[jamesknowsbest]

@kevin-liu-agriwebb
That error indicates that Device Farm failed to do a dry run on the tests, not that the annotation failed to execute.

Cannot instantiate class com.agriwebb.notebook2.test.LoginAndSyncTest`

The failure to parse and dry run the tests can happen for a few reasons from my experience

• There may be a syntax error or logic error in the tests(NPE for example)
• The tests reference an additional file outside of the test project. For example, the tests might reference an additional file on the desktop which the tests get parameters from. Device Farm's host machine will have no reference for it and the code referencing the file will fail during the dry run.
  • If this is the case, here are two ways I've found to reference additional files in Device farm.

If both of these possible issues are not the case, can you paste a code snippet from the LoginAndSyncTest test to help me reproduce the issue?

[kevin-liu-agriwebb]

@jamesknowsbest

I figured out it's because my LoginAndSyncTest inherits a parent class, and it seems the inheritance is not
honoured by device farm.

After removing inheritance from my tests, Test, BeforeTest and AfterTest annotations work well.

[sapins]

@kevin-liu-agriwebb
Are you able to run testng.xml on AWS device farm?

[jamesknowsbest]

@kevin-liu-agriwebb
Glad to hear the tests are working now. However, inheritance should be supported as this sample test project uses inheritance and is successful in Device Farm. I suspect that the class which LoginAndSyncTest extend from is doing one of the two things I mentioned earlier and that is causing the issue.

@sapins
The testng.xml file should be able to be used in Device Farm. It need to be added to the root of the jar file created in the deployment package however. This Pull request exemplifies how to add the file to the root of the jar.

TestNG docs: https://testng.org/doc/documentation-main.html#running-testng

This attribute should contain the path to a valid XML file inside the test jar (e.g. "resources/testng.xml"). The default is "testng.xml", which means a file called "testng.xml" at the root of the jar file. This option will be ignored unless -testjar is specified.

HTH
-James

[kevin-liu-agriwebb]

@sapins
Yes, I am. See the first comment of this thread for instructions on how-to.

[Gabrielstabile]

I'm facing some errors to execute my ios tests in aws farm, can someone help me?

[jamesknowsbest]

I'm facing some errors to execute my ios tests in aws farm, can someone help me?

Stackoverflow might be a better option than this github post for that. You can make a questions with the tag appium
https://stackoverflow.com/questions/ask

If there are specific questions about AWS Device Farm you can reach out for support on the aws forums
https://forums.aws.amazon.com/forum.jspa?forumID=193

Or you can open a support case if you have a support plan
https://console.aws.amazon.com/support/cases?region=us-east-1#/create