04/13/2017FeaturedChristopher Reichert

Designers, Product Managers, and other internal stakeholders can now participate in the discussion around a pull request because they can actually run the new fix or feature in a browser — just by visiting the Review app URL.

-- Heroku

Heroku Review Apps deployment reported to GitHub status check

Heroku's Review Apps are an exciting solution to continuous testing & delivery workflows. In a nutshell, Review Apps make it easy to test apps in the development cycle by creating a temporary testing environment every time a pull request is created on GitHub.

Review apps are the instant, disposable Heroku app environments that can spin up automatically with each GitHub pull request. They allow developers and their teams to automatically build and test any pull request, updated at every push, at a temporary, shareable URL.

Naturally, Review Apps are a great fit for automated post-deploy testing using Assertible which has first-class support for running tests as part of a continuous integration system. In this post, I will illustrate how to create an automated QA pipeline for a web service using GitHub, Heroku Review Apps, and Assertible.

Heroku Pending Review App deployment

Setup

I previously described at length how to configure Heroku Review Apps w/ GitHub and Assertible in my post Continuously testing Heroku GitHub deployments. The process for creating an automated QA pipeline can be reduced to 3 steps:

  1. Configure Heroku Review Apps w/ GitHub

    Heroku GitHub choose a repo

  2. Get an Assertible account and configure a web service

  3. Configure Assertible to receive deployment events from GitHub

    From you web service's overview page, navigate to the deployments tab and select the "GitHub integration" option. You should see a form to select a repository to monitor for deployment events.

    Assertible GitHub deployment integration setup

That's it! Now, anytime a new pull request is opened or commits are pushed to a PR, Assertible will receive deployment events and automatically run QA tests against every Heroku Review App created for that repository.

Goals

The primary goal of test automation is to improve productivity and save time. Our new automated QA pipeline acheives that by:

  • Testing each app deployment against the Heroku Review App unique URL automatically:

    This reduces the need to maintain code, write CI scripts, or other manual tasks that are error prone and take time away for your team. In the image below, a commit pushed to a PR is automatically built and deployed to our review app environment.

    Heroku Review App deployment pending on GitHub PR

  • Test results are received directly on GitHub pull request status checks

    There's no need to check and monitor another app's UI because developers will know the status of tests immediately after the code is deployed. There are a lot of bugs that can happen between the time your CI tests are executed and your app is deployed, hence the need for post-deploy testing.

    Heroku Review App deployment success status on GitHub pull request

  • Deployment history is tracked alongside test results

    Assertible logs each deployment with a version number and timestamp. This data can be correlated with failing test results to track regressions against specific versions of an application.

    Assertible deployment history for Heroku Review App

Enhancing the QA pipeline

Eventually, you will want to enhance your automated testing pipeline by expanding test coverage or integrating a manual QA process.

Integrating a manual QA processes:

An interesting approach to building out an automated QA pipeline is to integrate some form of manual testing. Obviously, someone should be manually testing new features after they are pushed.

The continuous testing pipeline illustrated in this post saves time by reducing the level of unecessary manual testing and other tasks. Ideally, manual testing only needs to be done for long workflows and features which don't have automated tests yet. Larger organizations may also have a formal QA process.

When automated tests fail, status checks are propogated to the pull request which indicate the problem clearly. No manual testing is needed for apps which have test failures. Futhermore, developers can link directly to the results of the test in Assertible to investigate failures.

Failing GitHub PR, no Heroku deployment, no Assertible tests run

When the automated QA tests all pass, Assertible sets the GitHub status check as successful. Manual QA testing can now be done on the app as part of an acceptance testing process, knowing that a wide range of QA has already been performed automatically on the live app.

Heroky Review App deployment going from fail to pass - request for manual QA

Expanding test coverage:

Another way to significantly enhance our QA pipeline is to add more automated test coverage. Assertible supports HTTP response data validation assertions for JSON APIs and HTML websites.

JSON API

For an API that serves JSON, the best option is to use the JSON Schema and JSON path data assertions. The image below shows the configuration view for multiple JSON assertions on a single test.

Assertible assertions for a JSON API

HTML Website

For a website that serves HTML, the HTML validation and Link checker assertions are an effortless way to validate behavior.

Assertible assertions for an HTML website

For more examples, check out the assertion documentation.


Start testing your web services today
Try Assertible for free. Have any questions? Contact us any time!


Conclusion

Building an automated QA pipeline using Assertible and Heroku Review Apps is practically turn-key. Review Apps make it easy to automate QA testing for web apps and APIs against unique environments which saves developer time, increases confidence when shipping code, and reduces bugs.

Resources

:: Christopher Reichert

Categories

Automate your QA pipeline

Define and test your web service with Assertible Track and test API deployments across environments Schedule API monitoring and failure alerts

Reduce bugs in web applications by using Assertible to create an automated QA pipeline that helps you catch failures & ship code faster.

Sign up for free