tl;dr - Learn how to run API integration tests against your web app directly on a CI machine using Assertible.

Check out the example on GitHub

In this post I'll illustrate how to run API integration tests during a CI build as part of a continuous testing strategy. The goal is to validate that the API or web application performs as expected under production-like circumstances. This strategy can be used to verify an app's functionality before a deployment or other parts of a continuous delivery processes.

Basically, ngrok is used to create a dynamic localhost tunnel to your app which is built and run on CI. The dynamic ngrok URL is passed to an Assertible Trigger which will run the API tests when executed. This testing technique is not specific to Python, Flask, or Travis CI and can be used from any continuous integration system or web application framework.

The setup is only four simple steps:

1. Create a Flask app
2. Create a Travis CI config
3. Create an Assertible web service
4. Copy the trigger URL to a Travis variable

This tutorial assumes you have a GitHub repository linked to Travis CI. If not, check the Travis CI tutorial for beginners

1. Create a Flask app

To get started, create a new directory for your code:

mkdir myapp && cd myapp

Save the following code to a file named app.py:

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello Assertible!"

if __name__ == "__main__":
    app.run()

Save the code and push it to a GitHub repository:

git add app.py
git commit -a -m "Checkin app.py"
git push

2. Create a Travis CI config

Next, create a file named .travis.yml and with the following content:

language: python
python:
  - "2.7"

addons:
  apt:
    packages:
      - ca-certificates

install: "pip install Flask"

script:
  - echo "Unit tests"

after_script:

  # start the web app
  - |
    python app.py &
    APP_PID=$!

  # download and install ngrok
  - curl -s https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip > ngrok.zip
  - unzip ngrok.zip
  - ./ngrok http 5000 > /dev/null &

  # sleep to allow ngrok to initialize
  - sleep 2

  # extract the ngrok url
  - NGROK_URL=$(curl -s localhost:4040/api/tunnels/command_line | jq --raw-output .public_url)

  # execute the API tests
  - |
    curl -s $TRIGGER_URL -d'{
      "environment": "'"$TRAVIS_BRANCH-$TRAVIS_JOB_NUMBER"'",
      "url": "'"$NGROK_URL"'",
      "wait": true
    }'

  - kill $APP_PID

This Travis configuration does several things after the code is built in the after_script section:

• Runs the app.py web app as a background process
• Downloads and install ngrok
• Starts ngrok pointed to app.py and saves the dynamic URL to NGROK_URL
• Calls an Assertible trigger with the dynamic NGROK_URL URL

Commit the .travis.yml file to the repository and push

git add .travis.yml
git commit -a -m "Checkin .travis.yml"
git push

3. Create an Assertible web service

Next, you need to create an Assertible account and create a new web service. If you don't have an Assertible account, simply sign-in to Assertible (it's free).

The first time you log-in, you will be prompted with a form to create a new web service. Enter the production URL where your web service will ultimately live or upload a Swagger spec or Postman Collection and click Import URL. This will show a preview of what will be created when you click Create service and tests.

NOTE: if you web service isn't in production yet, you can input any URL into the input

When you have verified the information, click Create service and tests:

4. Copy the trigger URL to a Travis variable

After you have created your web service in the Assertible dashboard, navigate to the web service's Settings tab and click Trigger URL in the left side navigation and copy the trigger URL to your clipboard.

In the Travis CI interface, navigate to your repositories build interface (see here for an example: https://travis-ci.org/assertible/python-example). On the right hand side, click the More options button then click Settings.

In the Environment Variables section, add a new variable named TRIGGER_URL and paste your Assertible trigger URL into the value input.

Finally, click Restart build or push some changes to your app to initiate a build. Navigate back to the Assertible web service dashboard and click Results. You should now see a result for your API integration tests with the dynamic ngrok URL:

Wrapping up

Continuous testing means validating your app at every staging of the continuous integration and delivery pipeline. Using the approach described in this post will allow you to reuse the same tests for post-deploy testing and scheduled monitoring. Additionally, defining tests in Assertible gives you access to powerful assertions like JSON Path and JSON Schema Validation as well as powerful integrations and automation like Slack and GitHub.

More resources and tutorials

• Python integration testing example
• Docs: Trigger URLs
• Running post deploy smoke tests from any Codeship pipeline
• Scheduled monitoring

Have any questions or comments? I'd love to hear it! Hit me up on Twitter or reach out directly.

:: Christopher Reichert