Why you should add API testing to your continuous integration pipeline

Automating deployments from continuous integration pipelines is considered a best-practice in the continuous integration community. It may sound risky to automate deployments, but continuous delivery has several major benefits:

• quick reaction times to bug fixes and feature releases
• reduces risk by automating repetitive time-consuming tasks
• improves productivity (again by automating repetitive tasks)
• lower effort required to ship releases

In an ideal world, every continuous delivery pipeline would incorporate automated functional testing after deployments.

Even if you are not automating deployments directly to production, there are several distinct advantages to automatically running integration, acceptance, and end-to-end tests against every release of your application immediately after it's deployed.

• Lower the burden and time cost of manual testing
• Validate components unreachable by unit tests
• Increased application test coverage
• Test exactly what the end-user sees (production)

Jump to Assertible's testing pipeline.

Lower the burden and time cost of manual testing

One of the biggest reasons to automate post-deploy integration testing in your continuous integration or delivery pipeline is to reduce the time and mental overhead needed to test redundant application features and regressions.

No sane team should be manually testing every feature every time it's deployed. Automating post-deploy application testing can make continuous delivery pipelines more robust in the same way unit testing enhances continuous integration pipelines.

Each new automated test you add to your suite represents a piece of functionality that never needs to be manually tested again. This can save teams a massive amount of time and allow QA testers and developers to focus on the current iteration of features or bug fixes.

Validate components unreachable by unit tests

In many cases, unit testing is preferred to integration or end-to-end testing. However, it's not always possible to write unit tests for every desired use-case. Writing and maintaining unit tests can have several downsides:

• Isolated unit tests can't validate behavior of integrated components.

• Small teams may not have the resources to develop complex testing harnesses or mocking needed to unit test everything.

• Unit tests cannot effectively test functionality from an end-user's perspective (unit tests are primarily for the developer).

In contrast, post-deploy integration tests can cover component interactions, workflows, etc

I recommend having a healthy amount of unit tests, integration tests, and end-to-end tests. A good starting point would be the Test Pyramid, which focuses on preferring unit tests when possible.

Increased application test coverage

Another major advantage of automated functional testing is increased test coverage of live environments. Because functional testing against live environments can test behavior that can't be tested at the unit-testing level, there is an overall increase in coverage when combining with unit tests (as illustrated by the Testing Pyramid above).

Once you have solid test automation in place, it should become easier to add new test cases for every feature and regression incrementally. Increasing the overall test coverage of your application becomes easier and encourages team members to add new tests frequently.

Test exactly what the end-user sees (production)

The only way to test the same thing your users see is to test your app in production. Naturally, many teams either manually test or build in-house scripts to validate a set of functionality against production (I know I have!).

The primary benefit to building out structured automated testing from your continuous delivery pipeline is that you can share the same test-suites across staging, QA, and production environments.

Having tests that automatically run against production from CI means that you will be the first to know when something goes wrong.

Things always go wrong in production, but this doesn’t have to be a bad thing. It’s an opportunity to learn about your system and the real world with which it interacts. With the right production monitoring tools and a good Continuous Delivery pipeline, you can build a set of feedback mechanisms that help you find out about issues as they happen and ship fixes quickly.

-- Rouan Wilsenach

Assertible's testing pipeline

At Assertible, we practice what we preach. Our testing pipeline is primarily comprised of three parts:

• Post-deploy integration testing staging environments

  As suggested in this blog, we deploy and test code every single time someone commits code. We manage staging environments that are linked to GitHub pull requests. When someone pushes code to a pull request, our automated test-suite is executed and the status is reported directly

  

• Post-deploy integration testing production

  When a pull request is merged and built, the code is deployed to production. We run the exact same sets of tests against our application the second it is deployed

  

• Scheduled Monitoring

  In between deployments, we also have continuous monitoring of our services on scheduled intervals.

Get started testing now!
Assertible is free to use. Contact us if you have any questions or feedback!

Examples and resources:

• Functional testing
• 3 preventable web app deployment failures.
• Techniques to reduce API testing errors and improve your QA process.
• 4 common API errors and how to test them.

:: Christopher Reichert