Apptest.ai Integration for Azure DevOps
With just a few lines of script, you can integrate Apptest.ai Service with Azure DevOps Pipeline to let our Testbot explore your apps and provide detailed reports.
To use the Apptest.ai service within Azure DevOps Pipeline, follow these two steps:
- Register the Apptest.ai Access Key.
- Add Apptest.ai to your build script (
azure-pipelines.yml).
Step 1: Adding the Apptest.ai Access Key to the Build Environment
- Copy the access key from the Profile Page in Apptest.ai.
- Register the APPTEST_AI_ACCESS_KEY variable in the Variable tab within Azure Pipeline settings.
- Once registered, click the lock icon to set the variable as secret, which prevents others from viewing it.
The format for APPTEST_AI_ACCESS_KEY should be set as follows:
Apptest.ai ID:Apptest.ai Access Key.For example:
jean@apptest.ai:ab8f3e321d631c84c9b1113.
Step 2: Calling the Script from azure-pipelines.yml
Add a script to execute the CI Tool Integration Script provided by Apptest.ai as follows. In this script, you need to pass the environment variable created earlier to the env section. Azure DevOps Pipeline requires explicit mapping for secret variables.
|
1 2 3 4 5 6 7 8 |
- bash: | git clone https://github.com/apptestai/ci_addon export binary_path=HackerNews.ipa export project_id='831' bash ./ci_addon/step.sh displayName: 'Tested by TestBot' env: APPTEST_AI_ACCESS_KEY: $(APPTEST_AI_ACCESS_KEY) |
At least two arguments—binary_path and project_id—are required to invoke the service.
- binary_path: Path to the binary you want to test (e.g.,
.apkfiles for Android,.ipafiles for iOS).- project_id: Project ID from Apptest.ai, which contains the settings for test time, device list, etc.
The project ID can be found on the Project Information page of Apptest.ai.
If you don’t want to download the CI integration script every time, you can include the
ci_addonrepository from Apptest.ai in your project repository in advance.If the project_id is also registered as a Pipeline variable, you can simplify your build YML file as follows:
|
1 2 3 4 |
- bash: bash ./ci_addon/step.sh displayName: 'Tested by TestBot' env: APPTEST_AI_ACCESS_KEY: $(APPTEST_AI_ACCESS_KEY) |
After committing the build script, Azure DevOps will initiate the build process. When the script reaches Apptest.ai, the test will be executed and results printed on the console.
Important Considerations
Currently, the Azure DevOps Pipeline has some errors while handling the output to the console.
- Logs that use ASCII color codes cannot be displayed during the build, but only after the build is finished.
- Additionally, there is an error that ignores the alignment setting through the
printfcommand in bash, causing the alignment of the result column to break.However, given the current evolution of Azure DevOps services, we hope these issues will be resolved soon.
The test results are stored in the test-results/apptest.ai/result.html with a detailed HTML report and an XML document in JUnit Test result format in test-results/apptestai/result.xml. You can save this file using the various deploy features provided by Azure DevOps. Below is an example of publishing the results as an artifact and test result in Azure.
If you want to change the location of the folder where the test results are stored, declare and pass the
test_result_pathenvironment variable.If you want to execute the next build script before finishing the test, declare the
waiting_for_test_resultsenvironment variable and set its value to “false”.
Finally, let me show you the complete azure-pipelines.yml example. The first case is for Android.
Complete azure-pipelines.yml Example
Android Case
Below is an example for an Android project:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
trigger: - master pool: vmImage: 'Ubuntu-16.04' steps: - task: Gradle@2 inputs: workingDirectory: '' gradleWrapperFile: 'gradlew' gradleOptions: '-Xmx3072m' javaHomeOption: 'JDKVersion' jdkVersionOption: '1.8' jdkArchitectureOption: 'x64' publishJUnitResults: false testResultsFiles: '**/TEST-*.xml' tasks: 'build' - bash: | git clone https://github.com/apptestai/ci_addon export binary_path=./app/build/outputs/apk/debug/app-debug.apk export project_id='831' bash ./ci_addon/step.sh displayName: 'Tested by TestBot' env: APPTEST_AI_ACCESS_KEY: $(APPTEST_AI_ACCESS_KEY) - task: PublishPipelineArtifact@0 inputs: artifactName: 'apptest-ai test result' targetPath: 'test-results/' - task: PublishTestResults@2 inputs: testResultsFormat: JUnit testResultsFiles: '**/*.xml' searchFolder: '$(System.DefaultWorkingDirectory)/test-results/' |
iOS Case
Next is an example for iOS:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
trigger: - master pool: vmImage: 'macOS-10.13' steps: - bash: echo -e "machine github.com\n login $CI_USER_TOKEN" >> ~/.netrc displayName: 'Setup build env' env: CI_USER_TOKEN: $(CI_USER_TOKEN) - bash: | pod install fastlane beta displayName: 'Build' env: MATCH_PASSWORD: $(MATCH_PASSWORD) - bash: | git clone https://github.com/apptestai/ci_addon export binary_path=HackerNews.ipa export project_id='831' bash ./ci_addon/step.sh displayName: 'Tested by TestBot' env: APPTEST_AI_ACCESS_KEY: $(APPTEST_AI_ACCESS_KEY) - task: PublishPipelineArtifact@0 inputs: artifactName: 'apptest-ai test result' targetPath: 'test-results/' - task: PublishTestResults@2 inputs: testResultsFormat: JUnit testResultsFiles: '**/*.xml' searchFolder: '$(System.DefaultWorkingDirectory)/test-results/' |
As you can see, the difference between Android and iOS cases lies in the actual building and packaging of the binary, but the process for calling the Apptest.ai testbot and saving the result is the same.
Final Notes
The test results, including a detailed HTML report (result.html) and an XML document in JUnit Test result format (result.xml), are stored in the test-results/apptestai/ directory. You can save these files using Azure DevOps’ deploy features, as shown in the examples above.
