You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Overview

In this tutorial, we will create a mobile test using Calabash and, optionaly, Xamarin Test Cloud.

In this case, the test (specification) is initialy created in JIRA as a Cucumber Test and afterwards is exported using the UI or the REST API.

 

Android and iOS

For the purpose of this tutorial, we will use an Android app as basis. However, the only change relevant for iOS would be to use "calabash-ios" command instead of the "calabash-android".

 

Requirements

  • Android SDK (or iOS SDK)
  • install the Ruby gems: "calabash-android" (or "calabash-ios"), "calabash-cucumber", "xamarin-test-cloud", "rubyzip"

Description

The code for our test will be the basic code that Android Studio generates, that is a basic "Hello World" application.

So the first step is to create create a Cucumber Test, of Cucumber Type "Scenario", in JIRA. The test will validate the presence of the "Hello World!" in the device's screen.

 

hello.feature
Feature: App startup banner

  @ABC-131
  Scenario: Message banner after startup
    When I wait for 2 seconds
    Then I see "Hello World!"

 

"calabash-android" provides some steps that can be reused in order to write Cucumber Scenarios/Scenario Outlines; in other words, if you just reuse those steps, you don't have to write any code at all for your tests.

 

After we create the Test in JIRA, and associate it with requirements, etc, you can export the specification of the test to a Cucumber .feature file (e.g. by REST API or by the UI action "Export to Cucumber" from within the Test Execution issue).

The created file will be similar to the above one but will contain the references to the Test issue key and the covered requirement issue key.

 

Project setup

In the project base folder, run the following command which will create a basic .feature.

 calabash-android gen

 

Note that the ApplicationManifest.xml must contain the "android.permission.INTERNET" permission. See example bellow.

 

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.example.smsf.myapplication">

    <application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:supportsRtl="true"
        android:theme="@style/AppTheme">
        <activity android:name=".MainActivity">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />

                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
    </application>

    <uses-permission android:name="android.permission.INTERNET" />
</manifest>



Running tests locally

You can run your tests locally using a device emulator or with a real connected device.

 calabash-android run app\build\outputs\apk\app-debug.apk  -x json -o data.json

 

In this case, the previous command will generate a Cucumber JSON report file that can be imported to Xray in the same way you would do for Cucumber tests in general (either by the REST API or through "Import Execution Results" action within the Test Execution).

 

Running tests in the Cloud with Xamarin Test Cloud

Before being able to submit the application, we must execute "calabash-android" with the "build" argument.

 calabash-android build app\build\outputs\apk\app-debug.apk

 

Before we submit the application for testing, we must know the API key on Test Cloud  (obtainable under the team's section) .

 

Afterwards, we can use the "test-cloud" utility in order to submit the packaged application (e.g. .apk), by identifying:

  • API key
  • Test Cloud's username (i.e. email)
  • the devices list ID
  • the series name

 

Cucumber config file used by the "test-cloud" utility:

config/cucumber.yml
android: PLATFORM=android -f json

 

Example syntax or invoking "test-cloud" for submission of the app:

 test-cloud submit app\build\outputs\apk\app-debug.apk a80654b0da3638a999a235e7b31d1433  --user tester@example.com  --devices 13684d00  --series "master" --locale "en_US" --async-json -p android --config config\cucumber.yml > test-cloud.json

 

As soon as the test run finishes in Xamarin's Test Cloud, the detailed report will be available in the Test Cloud's page.

It's possible to see some relevant numbers, such as the total devices  and tests with failures.

 

 

It's also possible to analyze a given test failure and see what step failed and the device and test logs.

 

 

Importing results to Xray

Note that Xray supports submission of a zip file containing multiple Cucumber JSON reports, one per each device.

In order to obtain Test Cloud's results in a machine friendly-way, we need to use Test Cloud's REST API since, currently, "test-cloud" unfortunately utility does not provide an immediate way of obtaining the results. 

However, it's possible to use the preliminary Test Cloud Ruby client SDK and API in order to produce a ZIP file containing the Cucumber JSON reports.

 

Bellow, is an example of such a potential implementation, where we pass the API key and the "test-cloud" output file as arguments, and that will produce a "results.zip" file.

 

obtain_tc_results.rb
require 'tmpdir'
require 'open-uri'
require 'zip'
require './client.rb'

def download_file(url, dir, filename)
    open("#{dir}/#{filename}", 'wb+') do |file|
        file << open(url).read
    end
end
api_key = ARGV[0]
testcloud_json_file = ARGV[1]
#api_key = "xxx"
#testrun_id = "xx"

testrun_id = JSON.parse(File.readlines(testcloud_json_file).last)["test_run_id"]
zipfile_name = "results.zip"

client = Xamarin::TestCloud::Api::V0::Client.new(api_key)
while !client.test_runs.results(testrun_id).finished
 puts "waiting for results..."
 sleep 5
end
results = client.test_runs.results(testrun_id)

File.unlink zipfile_name
tmp_dir = Dir.mktmpdir
begin
  results.logs.devices.each do |log_device|
      download_file(log_device.cucumber_json, tmp_dir, "#{log_device.device_configuration_id}.json")
  end
    input_filenames =  Dir.entries("#{tmp_dir}").select {|f| !File.directory? f}
    Zip::File.open(zipfile_name, Zip::File::CREATE) do |zipfile|
        input_filenames.each do |filename|
            zipfile.add(filename, tmp_dir + '/' + filename)
        end
    end

ensure
  FileUtils.remove_entry tmp_dir
end

 

Creating a zip file with all Cucumber JSON reports:

 obtain_tc_results.rb  a80654b0da3638a999a235e7b31d1111 test-cloud.json

 

After importing the results using the REST API, the execution screen details will provide information of the test run result grouped by device.

The "Context" section will be by filled with the name of the original Cucumber JSON report; in this case it contains the name/id of the device.


 

 

Learn more

Please see Testing with Cucumber for a high level overview on how to use Cucumber Tests with Xray.

 

References

 

  • No labels