Apptest.ai Integration for Azure DevOps: Automate Your App Testing
Integrating Apptest.ai with Azure DevOps Pipeline simplifies app testing by automating the process and providing detailed reports. A few simple configurations are all you need to get started.
Using Apptest.ai in Azure DevOps Pipeline
To use the Apptest.ai service within Azure DevOps Pipeline, follow these two steps:
- Register the Apptest.ai Access Key.
- Add the Apptest.ai build script to your
azure-pipelines.ymlfile.
Step 1: Adding the Apptest.ai Access Key to the Build Environment
- Start by copying your access key from the Profile Page in Apptest.ai.
- Go to the Variable tab in Azure Pipeline settings and add a new variable named
APPTEST_AI_ACCESS_KEY. - Secure the variable by clicking the lock icon to mark it as secret.
The
APPTEST_AI_ACCESS_KEYmust be formatted in the following way:Apptest.ai ID:Apptest.ai Access KeyExample:
jean@apptest.ai:ab8f3e321d631c84c9b1113
Step 2: Adding Apptest.ai to Your Build Script
To integrate Apptest.ai into your build pipeline, include the CI Tool Integration Script provided by Apptest.ai in your azure-pipelines.yml file. The script requires passing the environment variable created earlier into the env section. Azure DevOps Pipeline requires explicit mapping for secret variables.
Here’s an example:
|
1 2 3 4 5 6 7 8 |
- bash: | git clone https://github.com/apptestai/ci_addon export binary_path=HackerNews.ipa export project_id='831' bash ./ci_addon/step.sh displayName: 'Tested by TestBot' env: APPTEST_AI_ACCESS_KEY: $(APPTEST_AI_ACCESS_KEY) |
Required Parameters:
binary_path: Specifies the path to the binary file to be tested
(e.g.,.apkfor Android,.ipafor iOS).- project_id: The Project ID from Apptest.ai, which is used to retrieve settings such as test time, device list, and other configurations. This ID is available on the Project Information page in Apptest.ai.
If you prefer not to download the CI integration script every time, you can include the
ci_addonrepository in your project repository.Additionally, if the
project_idis stored as a pipeline variable, the YAML file can be further simplified:
|
1 2 3 4 |
- bash: bash ./ci_addon/step.sh displayName: 'Tested by TestBot' env: APPTEST_AI_ACCESS_KEY: $(APPTEST_AI_ACCESS_KEY) |
Once the build script is committed, Azure DevOps will initiate the pipeline. When the script communicates with Apptest.ai, tests will execute, and the results will be displayed in the console.
Important Considerations
There are some known limitations with Azure DevOps Pipeline output handling:
- Logs using ASCII color codes are not displayed during the build process but will appear after the build is complete.
- Alignment settings using the
printfcommand in Bash may cause misaligned columns in the results.Despite these limitations, Azure DevOps continues to evolve, and such issues are expected to improve in the future.
Managing Test Results
Test results are automatically saved in the test-results/apptest.ai/result.html file as a detailed HTML report. Additionally, an XML document in JUnit Test result format is stored in test-results/apptestai/result.xml.
You can save these files using the various deploy features provided by Azure DevOps. Below is an example of publishing the results as an artifact and test result in Azure.
If you want to change the location of the folder where the test results are stored, declare and pass the
test_result_pathenvironment variable.If you want to execute the next build script before finishing the test, declare the
waiting_for_test_resultsenvironment variable and set its value to"false".
Complete azure-pipelines.yml Example
Android Case
Below is an example for an Android project:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
trigger: - master pool: vmImage: 'Ubuntu-16.04' steps: - task: Gradle@2 inputs: workingDirectory: '' gradleWrapperFile: 'gradlew' gradleOptions: '-Xmx3072m' javaHomeOption: 'JDKVersion' jdkVersionOption: '1.8' jdkArchitectureOption: 'x64' publishJUnitResults: false testResultsFiles: '**/TEST-*.xml' tasks: 'build' - bash: | git clone https://github.com/apptestai/ci_addon export binary_path=./app/build/outputs/apk/debug/app-debug.apk export project_id='831' bash ./ci_addon/step.sh displayName: 'Tested by TestBot' env: APPTEST_AI_ACCESS_KEY: $(APPTEST_AI_ACCESS_KEY) - task: PublishPipelineArtifact@0 inputs: artifactName: 'apptest-ai test result' targetPath: 'test-results/' - task: PublishTestResults@2 inputs: testResultsFormat: JUnit testResultsFiles: '**/*.xml' searchFolder: '$(System.DefaultWorkingDirectory)/test-results/' |
iOS Case
Next is an example for iOS:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
trigger: - master pool: vmImage: 'macOS-10.13' steps: - bash: echo -e "machine github.com\n login $CI_USER_TOKEN" >> ~/.netrc displayName: 'Setup build env' env: CI_USER_TOKEN: $(CI_USER_TOKEN) - bash: | pod install fastlane beta displayName: 'Build' env: MATCH_PASSWORD: $(MATCH_PASSWORD) - bash: | git clone https://github.com/apptestai/ci_addon export binary_path=HackerNews.ipa export project_id='831' bash ./ci_addon/step.sh displayName: 'Tested by TestBot' env: APPTEST_AI_ACCESS_KEY: $(APPTEST_AI_ACCESS_KEY) - task: PublishPipelineArtifact@0 inputs: artifactName: 'apptest-ai test result' targetPath: 'test-results/' - task: PublishTestResults@2 inputs: testResultsFormat: JUnit testResultsFiles: '**/*.xml' searchFolder: '$(System.DefaultWorkingDirectory)/test-results/' |
As you can see, the difference between Android and iOS cases lies in the actual building and packaging of the binary, but the process for calling the Apptest.ai Testbot and saving the result is the same.
Final Notes
The test results, including a detailed HTML report (result.html) and an XML document in JUnit Test result format (result.xml), are stored in the test-results/apptestai/ directory. You can save these files using Azure DevOps’ deploy features, as shown in the examples above.
