Puma - Programmable Utility for Mobile Automation

Puma is a Python library for executing app-specific actions on mobile devices such as sending a message or starting a call. The goal is that you can focus on what should happen rather than how this should happen, so rather than write code to "tap on Bob's conversation on Alice's phone, enter the message, and press send", you can simply write "send a Telegram message to Bob from Alice's phone".

To execute actions on the mobile device, Puma uses Appium, and open-source project for UI automation.

Puma was created at the NFI to improve our process of creating test and reference datasets. Our best practices on creating test data can be found here. If you are wondering whether you should make your own test data, or whether you should use Puma for it, this document goes into the advantages of doing so.

Puma is an open-source non-commercial project, and community contributions to add support for apps, or improve support of existing apps are welcome! If you want to contribute, please read CONTRIBUTING.md.

📵 Puma currently only supports Android apps. iOS apps are on our to-do list!

Getting started

1. Install all required software (see the requirements section).
2. Connect your Android device (or start an emulator), make sure it is connected properly over ADB (See the section on troubleshooting if you encounter problems).
   • :warn: Make sure the phone is set to English, and all other requirements are met!
3. Get the UDID of your device by running adb devices (in the example below 954724ertyui74125 is the UDID of the device):

$ adb devices
> List of devices attached
954724ertyui74125  device

4. Install Puma. We recommend installing packages within a Python venv.

pip install pumapy

5. Run Appium. This starts the Appium server, a process that needs to run while you use Puma.

appium

Examples

If everything is setup correctly, you can now use Puma! Below are a few small examples to get started. If you want a more extensive step-by-step guide on how to use (and develop) Puma, please refer to the Puma Tutorial.

The code below shows a small example on how to search for the Eiffel Tower in Google Maps.

from puma.apps.android.google_maps.google_maps import GoogleMapsActions
from puma.utils import configure_default_logging

configure_default_logging()# Use Puma's logging configuration. You can also implement your own

phone = GoogleMapsActions("emulator-5444")
phone.search_place('eiffel tower')

This is a rather simple application, in the sense that it can be used without any form of registration. Other applications need some additional preparation, such as WhatsApp. For this application, you first need to register with a phone number. These kind of prerequisites are also described in the application README. The first time you use an application, there might be pop-ups explaining the app that Puma does not take into account, as these need to be confirmed only once. You need to do this manually the first time while running Puma. After registering, you can send a WhatsApp message to a contact with the code below:

from puma.apps.android.whatsapp.whatsapp import WhatsApp
from puma.utils import configure_default_logging

configure_default_logging() # Use Puma's logging configuration. You can also implement your own

alice = WhatsApp("<INSERT UDID HERE>")  # Initialize a connection with device
alice.create_new_chat(conversation="<Insert the contact name>",
                      first_message="Hello world!")  # Send a message to contact in your contact list
alice.send_message("Sorry for the spam :)")  # we can send a second message in the open conversation

An action might not always execute properly. If you want to verify that the action succeeded, you can add a special named argument to the action, which points to the function you want to verify the action with. We supply some commonly used verifications for users to use.

For example, verifying a Whatsapp message has been sent:

app = WhatsApp('<INSERT UDID HERE>')
app.send_message(conversation='Bob', message_text='Sorry for the spam :)', verify_with=app.is_message_marked_sent)

This will verify if the expected message has indeed been sent. If not, it will log this using the Ground Truth logger. It will not stop the execution of the following steps. For more information, see the action documentation.

Congratulations, you just did a search query in Google Maps and/or sent a WhatsApp message without touching your phone! You can now explore what other functions are possible with Puma in WhatsApp, or try a different application. You could even start working on adding support for a new app.

Supported apps

The following apps are supported by Puma. Each app has its own documentation page detailing the supported actions with example implementations:

• Google Camera
• Google Chrome
• Google Maps
• Google Play Store
• Open Camera
• Snapchat
• Telegram
• TeleGuard
• WhatsApp
• WhatsApp for Business

Right now only Android is supported.

To get a full overview of all functionality and pydoc of a specific app, run

# Example for WhatsApp
python -m pydoc puma/apps/android/whatsapp/whatsapp.py

Supported versions

The currently supported version of each app is mentioned in the documentation above (and in the source code). When Puma code breaks due to UI changes (for example when app Xyz updates from v2 to v3), Puma will be updated to support Xyz v3. This new version of Puma does will only be tested against Xyz v3: if you still want to use Xyz v2, you simply have to use an older release of Puma.

To make it easy for users to lookup older versions, git tags will be used to tag app versions. So in the above example you'd simply have to look up the tag Xyz_v2.

If you are running your script on a newer app version than the tag, it is advised to first run the test script of your app (can be found in the test scripts directory). This test script includes each action that can be performed on the phone, and running these first will inform you if all actions are still supported, without messing up your experiment.

Navigation

You need to be careful about navigation. For example, some methods require you to already be in a conversation. However, most methods give you the option to navigate to a specific conversation. 2 examples:

Example 1

from puma.apps.android.whatsapp.whatsapp import WhatsApp
from puma.utils import configure_default_logging

configure_default_logging() # Use Puma's logging configuration. You can also implement your own
alice = WhatsApp("emulator-5554")  # initialize a connection with device emulator-5554
alice.go_to_state(WhatsApp.chat_state, conversation="Bob")
alice.send_message("message_text")

In this example, the message is sent to the conversation with "Bob", by selecting the conversation manually. The second example below automates this step.

Example 2

from puma.apps.android.whatsapp.whatsapp import WhatsApp
from puma.utils import configure_default_logging

configure_default_logging() # Use Puma's logging configuration. You can also implement your own
alice = WhatsApp("emulator-5554")  # initialize a connection with device emulator-5554
alice.send_message("message_text", conversation="Bob")
alice.send_message("message_text2")

In the second example, the chat conversation to send the message in is supplied as a parameter. Before the message is sent, there will be navigated to the home screen first, and then the chat "Bob" will be selected.

Note that the second message is sent to the current chat conversation. Puma will detect that a conversation was already opened, so it will not navigate back to the main screen and reopen the same conversation.

Requirements

Install dependencies

First off, run the installation scripts in the install folder. See the installation manual for more details.

Android Device(s) or Emulators

You can either use a physical Android device or an Android emulator. See Optional: Android Studio for instructions on installing Android Studio and running an emulator

• Have the Android device(s) or emulator(s) connected to the system where Puma runs, configured as follows:
  • Connected to the Internet
  • Language set to English
  • File transfer enabled
  • (Root access is not needed)

You can check if the device is connected:

adb devices
  > List of devices attached
894759843jjg99993  device

If the status says device, the device is connected and available.

Optional: Android Studio (for running an emulator)

For more information about Android Emulators, refer to the Android developer website Follow these steps to create and start an Android emulator:

1. Install Android Studio.
2. Create an Android Virtual Device (avd) We recommend a Pixel with the Playstore enabled, and a recent Android version. For running 1 or a few apps, the default configuration can be used.
3. Start the emulator.

If you want to run the emulator from the commandline, refer to Start the emulator from the command line.

Optional: OCR module

Puma has an OCR module which is required for some apps. See the documentation of the apps you want ot use whether you need OCR.

Top use the OCR module you need to install Tesseract:

sudo apt install tesseract-ocr

Or use the Windows installer.

Optional: FFMPEG

To use video_utils.py you need to install ffmpeg:

sudo apt install ffmpeg

This utils code offers a way to process screen recordings (namely concatenating videos and stitching them together horizontally).

Logging in Puma

Puma uses Python’s standard logging library.

Default Behavior

• As a CLI or main module: Puma configures default logging so INFO and higher messages are visible.
• In Jupyter notebooks: Puma enables default logging so logs are visible in notebook cells.
• As a module in another project: Puma does not configure logging; messages are only shown if your application configures logging.

Ground Truth Logging

Puma contains a separate 'Ground Truth' logger (GTL), which logs all actions and navigation steps that are performed on a device during a Puma run. These logs are stored in separate log files with the _gtl suffix. The log lines produced by this logger are also present in the regular log files, but the GTL logs only contain information about actions on a device.

How to See Puma’s Logs

To see Puma logs in your own script, opt-in to Puma's default log format and level by calling:

from puma.utils import configure_default_logging
configure_default_logging()

This is also shown in the examples above. If you want to use your own logging format, you can configure Python logging (e.g., with logging.basicConfig(level=logging.INFO)).

Troubleshooting

ADB shows status unauthorized

This happens when you did not allow data transfer via usb. Tap on the charging popup or go to settings > USB preferences and select File Transfer.

Adb device cannot connect

If the status of your device is unauthorized, make sure USB debugging is enabled in developer options:

• Enable developer options
• Enable USB debugging
• Connect your device to your computer, open a terminal and run adb devices
• Your device should now show a popup to allow USB debugging. Press always allow.

If you do not get the pop-up, reset USB debugging authorisation in Settings > Developer options > Revoke USB debugging authorisations and reconnect the device and run adb devices again.

Android Emulator won't start in Android Studio

We have encountered this in MacOS, but it could also occor on other platforms. If you encounter an issue where the Android Emulator won't start, it might be due to the location where Android Studio installs system images. By default, Android Studio installs system images in the $HOME/Library/Android/Sdk directory. However, our configuration may expect the SDK to be located in a different directory, such as $HOME/Android/Sdk.

A workaround is to create a symbolic link:

ln -s $HOME/Library/Android/Sdk/system-images $HOME/Android/Sdk/system-images

Installing Appium with npm fails

If you are behind a proxy and the appium install hangs, make sure to configure your ~/.npmrc with the following settings. Fill in the values, restart terminal and try again:

registry=<your organization registry>
proxy=<organization proxy>
https-proxy=<organization proxy>
http-proxy=<organization proxy>
strict-ssl=false

Alternatively, you can also download Appium Desktop, make the binary executable and start it manually before running Puma.

sudo chmod +x Appium-Server-GUI-*.AppImage
./Appium-Server-GUI-*.AppImage

• Do not change the default settings
• Click the startServer button
• Now you can run Puma

ConnectionRefusedError: [Errno 111] Connection refused

This error is probably caused by Appium not running. Start Appium first and try again.

Appium Action fails due to popup

When first using the app, sometimes you get popups the first time you do a specific action, for instance when sending a view-once photo. Because this only occurs the first time, it is not handled by the code. The advice is when running into this problem, manually click Ok on the pop-up and try again. To ensure this does not happen in the middle of your test data script, first do a test run by executing the test script for your application.

My application is not present on the device

Install the APK on the device you want to use. When using an emulator, this can be done by dragging the APK file onto the emulator, this automatically installs the APK on the device. For physical devices as well as emulators, you could use adb install. See the developer docs

Pop-ups break my code!

Some applications have pop-ups which appear the first time that the application is opened. Puma does not handle these pop-ups, these should be manually clicked once to remove them. The same holds for pop-ups that request permissions, these should be manually clicked. Note: If your app has other pop-ups that happen regularly, Puma should support these.