Designers, Product Managers, and other internal stakeholders can now participate in the discussion around a pull request because they can actually run the new fix or feature in a browser — just by visiting the Review app URL.

-- Heroku

Heroku's Review Apps are an exciting solution to continuous testing & delivery workflows. In a nutshell, Review Apps make it easy to test apps in the development cycle by creating a temporary testing environment every time a pull request is created on GitHub.

Review apps are the instant, disposable Heroku app environments that can spin up automatically with each GitHub pull request. They allow developers and their teams to automatically build and test any pull request, updated at every push, at a temporary, shareable URL.

Naturally, Review Apps are a great fit for automated post-deploy testing using Assertible which has first-class support for running tests as part of a continuous integration system. In this post, I will illustrate how to create an automated QA pipeline for a web service using GitHub, Heroku Review Apps, and Assertible.

Setup

I previously described at length how to configure Heroku Review Apps w/ GitHub and Assertible in my post Continuously testing Heroku GitHub deployments. The process for creating an automated QA pipeline can be reduced to 3 steps:

1. Configure Heroku Review Apps w/ GitHub

   

   
   

2. Get an Assertible account and configure a web service

3. Configure Assertible to receive deployment events from GitHub

   From you web service's overview page, navigate to the deployments tab and select the "GitHub integration" option. You should see a form to select a repository to monitor for deployment events.

   

   
   

That's it! Now, anytime a new pull request is opened or commits are pushed to a PR, Assertible will receive deployment events and automatically run QA tests against every Heroku Review App created for that repository.

Goals

The primary goal of test automation is to improve productivity and save time. Our new automated QA pipeline acheives that by:

• Testing each app deployment against the Heroku Review App unique URL automatically:

  This reduces the need to maintain code, write CI scripts, or other manual tasks that are error prone and take time away for your team. In the image below, a commit pushed to a PR is automatically built and deployed to our review app environment.

  

  
  

• Test results are received directly on GitHub pull request status checks

  There's no need to check and monitor another app's UI because developers will know the status of tests immediately after the code is deployed. There are a lot of bugs that can happen between the time your CI tests are executed and your app is deployed, hence the need for post-deploy testing.

  

  
  

• Deployment history is tracked alongside test results

  Assertible logs each deployment with a version number and timestamp. This data can be correlated with failing test results to track regressions against specific versions of an application.

  

  
  

Enhancing the QA pipeline

Eventually, you will want to enhance your automated testing pipeline by expanding test coverage or integrating a manual QA process.

Integrating a manual QA processes:

An interesting approach to building out an automated QA pipeline is to integrate some form of manual testing. Obviously, someone should be manually testing new features after they are pushed.

The continuous testing pipeline illustrated in this post saves time by reducing the level of unecessary manual testing and other tasks. Ideally, manual testing only needs to be done for long workflows and features which don't have automated tests yet. Larger organizations may also have a formal QA process.

When automated tests fail, status checks are propogated to the pull request which indicate the problem clearly. No manual testing is needed for apps which have test failures. Futhermore, developers can link directly to the results of the test in Assertible to investigate failures.

When the automated QA tests all pass, Assertible sets the GitHub status check as successful. Manual QA testing can now be done on the app as part of an acceptance testing process, knowing that a wide range of QA has already been performed automatically on the live app.

Expanding test coverage:

Another way to significantly enhance our QA pipeline is to add more automated test coverage. Assertible supports HTTP response data validation assertions for JSON APIs and HTML websites.

JSON API

For an API that serves JSON, the best option is to use the JSON Schema and JSON path data assertions. The image below shows the configuration view for multiple JSON assertions on a single test.

HTML Website

For a website that serves HTML, the HTML validation and Link checker assertions are an effortless way to validate behavior.

For more examples, check out the assertion documentation.

Start testing your web services today
Try Assertible for free. Have any questions? Contact us any time!

Conclusion

Building an automated QA pipeline using Assertible and Heroku Review Apps is practically turn-key. Review Apps make it easy to automate QA testing for web apps and APIs against unique environments which saves developer time, increases confidence when shipping code, and reduces bugs.

Resources

• Heroku Review Apps docs
• Why we test every API deployment
• Continuously testing Heroku GitHub deployments.
• Running Acceptance with Heroku Review Apps

:: Christopher Reichert