Skip to main content

End to End Testing Automation

Overview​

In order to scale and accelerate the end-to-end testing of our Native Apps, we integrated the Maestro E2E testing framework into our Gitlab CI workflow automation pipelines.

The pipelines run the E2E tests on virtual devices and generate test reports that are uploaded to the GitLab repository, linked in merge requests, and posted to our dedicated E2E testing notifications Slack channel.

Running the E2E Tests in Gitlab CI​

The E2E tests are run in the GitLab CI by passing certain environment variables to the pipelines that are scheduled or ran manually ad-hoc.

The environment variables are:

  • IS_MAESTRO_ANDROID_RUN: Setting this to true will trigger the pipeline to spawn the E2E Tests Android jobs.
  • IS_MAESTRO_IOS_RUN: Setting this to true will trigger the pipeline to spawn the E2E Tests iOS jobs.
  • TEST_TAGS: This variable is used to filter the E2E tests that are run. It can be used to run only the tests that are tagged with the given tag. For example, TEST_TAGS=account will run only the tests that are tagged with account in the E2E test files. Multiple tags can be passed by separating them with a comma, e.g. TEST_TAGS=account,login.

A Pipeline can be triggered manually by clicking the Run Pipeline button in the GitLab UI, which will take you to the Run Pipeline page.

A Pipeline can be scheduled to run at a specific time on the Pipeline Schedules page.

A Pipeline given the above environment variables will spawn E2E jobs for the given platforms and will run tests for the given test tags, for example:

Architecture of the E2E Test Pipeline​

The E2E test pipeline is made up of several components, which are the:

  • GitLab CI workflow runners that execute the pipeline jobs on the GitLab CI/CD platform through our Kubernetes Cluster
  • GitLab CI configuration file that defines
    • the jobs to execute in the pipeline,
    • stages to group the various jobs,
    • environment variables used to configure the jobs, and
    • rules to define when the jobs are executed
  • Container Images used to configure the different system environments needed for the iOS and Android E2E test pipeline jobs to run in, with required dependencies and tools pre-installed
  • Shell Scripts that are used to automate the setup and execution of the E2E tests in the pipeline jobs themselves (reducing the complexity of the Gitlab CI configuration file itself).

Architecture visualisation​

E2E Test Pipeline Architecture

Gitlab CI​

info

If you are not familiar with GitLab CI, please read our GitLab CI/CD documentation first.

Additional information can be found at the official Gitlab CI documentation.

Gitlab CI is the tool we use to automate our CI/CD workflows, such as building and deploying our apps, running unit tests, and running E2E tests without having to do it manually.

Our project's Gitlab CI configuration is stored in the .gitlab-ci.yml file.

This file is read by Gitlab CI each time a pipeline is triggered by a commit, merge request, or scheduled job, and the jobs defined in the file are executed.

Gitlab CI Configuration for E2E Jobs​

The E2E jobs are all run in the Test stage of the pipeline, and are executed in parallel for iOS, and sequentially for Android.

info

Currently, the Android jobs can only run one at a time whilst using the legacy "Mac Mini 2" hardware. See the Android E2E Test job environment and hardware section.

CI Job Tags​

E2E Test jobs for different platforms used different tags that tell Gitlab CI which runners to use to execute the jobs.

  • For iOS: tart
    • If testing changes to the TartVM container image, use jobs with the tart-testing tag.
  • For Android: mac-mini-2
    • Once Android E2E Test jobs are migrated to use the EC2 Baremetal instances, the jobs should be tagged using the baremetal tag, with a runner t-shirt size variant to determine the memory required, e.g. baremetal-xlarge.

CI Job Rules​

Each Job only runs if the given rules are met, which are defined in the rules_templates section of the Gitlab CI configuration file. For example, if the IS_MAESTRO_ANDROID_RUN or IS_MAESTRO_IOS_RUN environment variables are set to true, the E2E jobs will run. There is an opposite rule that is applied to all other CI jobs, to ensure that only E2E Test jobs are run for E2E Test pipelines.

CI Job Templates​

The E2E jobs use job templates for iOS and Android, e2e_ios_test_template and e2e_android_test_template respectively. This allows us to define the common configuration for the jobs, such as the image and tags to use, and the scripts to be executed.

The actual E2E job declaration then extends these templates and declares variables for BRAND and PLATFORM based on which job we want to run.

These variables are then used in the shell scripts to configure the environment and run the E2E tests themselves.

Kubernetes Cluster and Container Images​

info

If you are not familiar with Kubernetes, please read this Kubernetes Cluster overview first.

To run our automated CI/CD workflows, we use a Kubernetes Cluster, which orchestrates, or schedules and manages, the virtual and physical machines in our Cloud infrastructure that we use to run the pipeline jobs created by Gitlab CI.

We run our GitLab CI jobs in 'containers' - which are pre-defined virtualised environments that set up any required dependencies and tools needed to run the pipeline jobs in the container.

Each container is configured via a container image file, e.g. Dockerfile for our Docker container image used for Android, or Packer for our TartVM container image used for iOS.

The container images are stored in our Containers repository.

note

Some of the configuration and dependencies for our E2E jobs are stored in the container images themselves, and some are stored in the GitLab CI configuration and shell scripts.

This is done currently so that the E2E test workflows can be run locally on a host machine, and not just in the GitLab CI environment, as a result of needing to run the Android E2E tests on the legacy "Mac Mini 2" which is not a virtualised environment.

Once this is no longer needed, the container images should be updated to contain all the required dependencies and configurations, and the GitLab CI configuration and shell scripts should be updated to only contain the commands to run the E2E tests where possible.

Container Image for Android​

To automate our Android app builds and E2E tests efficiently, we use this AndroidSDK Docker image.

It is forked from this public AndroidSDK GitHub project.

This image can be used by declaring

image: 285654214828.dkr.ecr.ap-southeast-2.amazonaws.com/android-sdk:latest

in a GitLab CI job, in the .gitlab-ci.yml file.

Maintaining the AndroidSDK container image​

Anyone can modify the container image (e.g. the Dockerfile) by committing and pushing changes to the repository. Any pushed commits to the local branch will trigger a new build of the image, and the image will be published, and usable in the GitLab CI jobs.

danger

This container image is used for both Android Build and Android E2E testing jobs, and there is currently no implementation to create variants for the two use cases, and there is no documented workflow to test changes to the container image locally.

All commits will trigger a new container image build and publish action. This means that a change that introduces a bug in the container image will affect both Build and E2E Test jobs. Please be careful when making changes to the container image.

note

TODO: Enlist Cloud teams help to write guide on setting up the required tooling to test changes to the container image locally.

Container Images for iOS​

For our iOS Build, Deploy and E2E Test jobs in Gitlab CI, we use virtualised macOS machines hosted on our own Apple hardware in our cloud infrastructure.

The technology we use to virtualise MacOS is TartVM, which allows us to setup and configure the virtual macOS with all the required dependencies and tools, and then cache the image for faster CI jobs.

The TartVM image is stored in our Macos Image Templates repository, which is a fork of the public GitHub repository of the same name, hosted by Cirrus Labs who created and maintain TartVM.

The TartVM image is usable in Gitlab CI jobs by extending the .ios_vm_setup template job in any Gitlab CI job, in the .gitlab-ci.yml file, or by appending the specific image and tags to a given job:

extends: .ios_vm_setup

or

image: ventura-entain:14.2
tags:
  - tart

Note: in ventura-entain:14.2, ventura-entain is the name of the TartVM image pointing to a specific macOS version, whereas 14.2 is the version of Xcode needed.

Maintaining the TartVM image​

Anyone can maintain the TartVM image by committing and pushing changes to the repository.

Unlike the AndroidSDK container image project, the TartVM repository does not have any CI jobs of its own.

So any changes committed and pushed to the repository will not trigger a new build and publish action, and instead the Cloud Team should be contacted to deploy the new image to our self-hosted Apple hardware, to be used in the GitLab CI jobs.

To test changes locally, you will need to install HashiCorp's Packer, and follow the instructions in the README of the Macos Image Templates repository as well as the TartVM documentation.

To test the changes to the TartVM image work in our E2E CI test jobs, you will need to ask the Cloud team to deploy the committed changes to a dedicated M2 Mac Mini machine used for testing, which can be scheduled to run a job using the tart-test tag in our CI job configuration.

danger

This container image is used for both iOS build, deploy and E2E test jobs. There are currently no variants for the two use cases, and there is no documented workflow to test changes to the container image locally (TODO - ask Cloud Team for help).

When making changes to the TartVM image, be sure to either create and deploy a separate image for testing, or to test the changes on tart-testing jobs first, before asking the Cloud team to deploy the new image to the rest of the M2 Mac Minis used to build and deploy our iOS app.

Android E2E Test job environment and hardware​

Presently, the only way to run E2E Tests for Android is to use the legacy "Mac Mini 2" hardware. This is due to the regular EC2 Instances and TartVM not supporting virtualisation, which is required to run the Android Emulator. As a result, we can only run one (1) Android E2E Test job at a time, sequentially.

Work is in progress to migrate the Android E2E tests to run on EC2 Baremetal instances, which support virtualisation, and will allow us to run the Android Emulator in a virtualised environment, and run the E2E tests in parallel.

The Mac Mini 2 - Legacy Android E2E Testing Environment​

The Mac Mini 2 is a physical machine that is hosted in our Cloud infrastructure, and is used to run the Android E2E tests in GitLab CI.

It is the only machine that can run the Android Emulator for the E2E tests in CI, and is a physical machine, not a virtualised environment, meaning that it needs to clean up the virtual devices after each job is run.

The Mac Mini 2 is configured with the required dependencies and tools to build the Android app, run React Native, create and start and emulator and run the E2E tests. Note that the E2E Shell Scripts will nevertheless check whether app required dependencies and tool versions for the Android Emulator, Android App build and E2E tests are present, and try to install them if it cannot find them.

Gitlab Runner on the Mac Mini 2​

A ShellExecutor is used to run the jobs on the Mac Mini 2.

The Gitlab Runner itself is installed via Brew, and is configured to run as a background service on the Mac Mini 2 when started or restarted.

You can check whether the background service is running by SSHing into the Mac Mini 2 and running the following command:

brew services list

which should output something like:

Name    Status  User     Plist
gitlab-runner started root     /Library/LaunchDaemons/homebrew.mxcl.gitlab-runner.plist

If the service is not running, you can start it by running the following command:

brew services start gitlab-runner

Starting the Gitlab Runner as a foreground process​

If for any reason the background service is not working, you can fall back to running the gitlab-runner as a foreground process in a terminal window.

To do this, SSH into the Mac Mini 2, and run the following command:

gitlab-runner start

This will start the Gitlab Runner as a foreground process, and you will be able to see the logs of the jobs that are being run.

To stop the Gitlab Runner, press Ctrl + C in the terminal window.

The terminal window must remain open for the Gitlab Runner to continue running.

Maintaining the Mac Mini 2​

The Mac Mini 2 hardware, the physical machine, is managed and maintained by the Desktop Support team.

For any configuration issues within the MacOS environment or issues with the Gitlab Runner, contact the Cloud Support team.

Any developer should be able to SSH into the Mac Mini 2 to maintain any of the required system-installed dependencies or tools, or to check the status of the gitlab-runner service.

If you need to update any dependencies or tools installed via Brew, you must first stop the gitlab-runner service, and then restart it after you have finished updating the dependencies or tools.

To do this, SSH into the Mac Mini 2, and run the following commands:

# Stop the gitlab-runner service
brew services stop gitlab-runner
# Update the dependencies or tools
brew upgrade
# Restart the gitlab-runner service
brew services start gitlab-runner

If Android E2E Tests are not running in GitLab CI, or the Mac Mini 2 is unavailable​

On rare occasion, the Mac Mini 2 may become unavailable, and the Android E2E tests will fail to run in the GitLab CI.

If this happens, please contact the Desktop Support team to investigate and resolve the issue.

Should the Mac Mini 2 have restarted or shut down, Desktop Support will need to login to the machine before anyone can SSH into it remotely to start or check the gitlab-runner service, for the Android E2E tests to run again.

Android E2E Tests on EC2 Baremetal Instances​

Work is in progress to migrate the Android E2E tests to run on EC2 Baremetal instances, which support virtualisation, and will allow us to run the Android Emulator in a virtualised environment, and run the E2E tests in parallel.

Things to note about the EC2 Baremetal instance:

  • We only have 1 provisioned
  • It costs about 5x more than our other EC2 instances, but has much more power (vCPU, RAM, memory)
  • One instance can run several E2E test jobs in parallel.
  • Due to the cost, it is kept in a stopped state when not in use, and is only started when needed for E2E tests
  • CI Jobs running on this instance will stay "Pending" until the instance is started, and will then be picked up by the Gitlab Runner and executed.

We use our custom AndroidSDK container image on this instance to run the E2E tests, the same one used to build the Android app bundle that gets published to the Google Play Store.

For more details on the container image go back to the above section, Container Image for Android.

For any issues or questions regarding the EC2 Baremetal instance, please contact the Cloud Support team.

iOS E2E Test job environment and hardware​

The iOS E2E tests are run on virtualised macOS machines hosted on our own Apple hardware in our cloud infrastructure.

We have 10 M2 Mac Mini machines, each capable of running 2 CI jobs at a time, in a virtualised macOS environment, using TartVM, providing 20 concurrent jobs in total we can run for our iOS build, deploy and E2E test jobs.

The TartVM image is stored in our Macos Image Templates repository. For more details on the TartVM image, go back to the above section, Container Images for iOS.

For any issues or questions regarding the TartVM image or M2 Mac Minis used for the TartVM runners, please contact the Cloud team.

Shell Scripts​

Several custom shell scripts are used in the GitLab CI jobs to automate the setup and execution of the E2E tests.

They are executed inside of the before-script and script sections of the iOS and Android E2E test job templates, in our Gitlab CI Configuration file.

Using shell scripts instead of defining the commands directly in the Gitlab CI configuration file allows us to lint the shell scripts for errors and warnings via tools like ShellCheck, as well as enabling us to maintain and test the commands locally on a host machine, and not just in the GitLab CI environment.

When run in a CI environment, these scripts will:

  • Start the Metro Bundler with E2E files bundled, in a background process to not block the rest of the CI job execution.
  • Check that the required dependencies are installed, e.g. Maestro CLI
  • Create and start the iOS Simulator or Android Emulator, in a background process to not block the rest of the CI job execution.
  • Run the E2E enabled App on the Simulator or Emulator
  • Run the E2E tests for the given TEST_TAGS, passed as an environment variable to the pipeline.
  • Collect any test reports generated by the E2E tests, and upload them to the GitLab repository to be linked in the merge request or the pipeline artifacts.
  • Send a notification to the dedicated E2E testing notifications Slack channel, with a link to the test reports and the given CI pipeline job.
  • On the tests finishing, or any error in the E2E shell scripts, run a cleanup script to kill the Metro Bundler process, which will remain running idle otherwise - so that the CI job process can exit.

The scripts are located in the e2e/scripts directory of the Entain Native Apps repository:

  • prepare-merge-request-e2e-tests.sh: Allows us to check only changed E2E test files in merge request pipelines, by copying the changed files into the folder structure required by Maestro temporarily.
  • launch_e2e_bundler.sh: Starts the Metro bundler for the E2E tests.
  • run_tests.sh: This script runs the E2E tests on the virtual devices. It executes the necessary commands to
    • install the required dependencies, or install specific versions of the dependencies based on environment variables.
    • setup and start the virtual devices (e.g. emulator or simulator)
    • run the E2E tests
    • generate the test reports
  • post_run_cleanup.sh: Used to clean up the virtual devices after the E2E tests are completed, if the scripts are run locally or on a host machine/non-container environment.
  • send_slack_report.sh: Used to send the test reports as notifications to the Slack channel.
danger

These scripts are written using POSIX shell syntax, using only POSIX-compatible commands, to ensure that they can be run in both MacOS and Linux environments, in any shell (e.g. Bash, Dash, Zsh, etc.).

Please ensure you have installed and set up ShellCheck in either your IDE, Code Editor or terminal to be able to lint the shell scripts to ensure that the POSIX syntax is correct, and that there are no errors or warnings in the scripts that would make the script not POSIX-compatible.

Moving environment setup logic to Container Images​

Once the E2E tests for Android are migrated from the legacy "Mac Mini 2" to EC2 Baremetal instances, the environment setup logic that does not need to be configured dynamically via environment variables for every CI job run should be moved to the AndroidSDK and TartVM container images.

This will reduce the complexity of the E2E CI jobs architecture and speed up the CI job execution time, as the container image environment setup and dependency installation can be cached, whereas running these setup commands in the Shell scripts requires that they be executed and dependencies downloaded and installed for every CI job run.

Things that can be moved to the container images are:

  • Installing the required dependencies for the Android Emulator, Android App build and E2E tests
  • Installing specific versions of the dependencies based on environment variables
  • Setting up and creating the virtual devices (e.g. emulator or simulator)

Different variants of container images could be created if multiple versions of Android and iOS are needed, and if different variants of virtual devices are to be tested. The variant container images can then be used in the GitLab CI jobs by appending the specific image and tags to a given job:

image: ventura-entain:14.2
# or
image: sonoma-entain:15.0

Things that cannot be moved to the container image and which need to stay in the shell scripts or the Gitlab CI configuration file:

  • Starting the Metro Bundler
  • Starting the virtual devices (e.g. emulator or simulator)
  • Running the E2E tests
  • Generating the test reports

Testing and Maintaining the E2E Shell Scripts locally​

The E2E tests can be run locally using the scripts in the e2e/scripts directory.

In a terminal, navigate to the e2e/scripts directory, and run the following commands:

# Install the dependencies
yarn
# Start the Metro Bundler
BRAND=ladbrokes sh ./launch_e2e_bundler.sh

Then, in a separate terminal, run the following commands:

# Install the dependencies for iOS (optional)
yarn pod
# Run the E2E tests for iOS
BRAND=ladbrokes PLATFORM=ios TEST_TAGS=account sh ./run_tests.sh

and similarly for Android

# Run the E2E tests for Android
BRAND=ladbrokes PLATFORM=android TEST_TAGS=account sh ./run_tests.sh

When run locally, these scripts will:

  • Start the Metro Bundler with E2E files bundled
  • Check that the required dependencies are installed, i.e. Maestro
  • Create and start the iOS Simulator or Android Emulator
  • Run the E2E enabled App on the Simulator or Emulator
  • Run the E2E tests for the given TEST_TAGS

If any errors occur, the post_run_cleanup.sh script will run automatically, killing any running Metro Bundler, Simulator or Emulator processes, and cleaning up any virtual devices that were created and their temporary files.