This post will walk through setting up a continuous integration, deployment, and testing pipeline for a Node.js app using CircleCI, Heroku, and Assertible.
Continuous integration is large part of modern development workflows. Most of us are familiar with basic CI services that automatically build code as new changes are pushed. But what if you took that a step further and not only built your code, but also deployed and tested it within the same pipeline? This is continuous development.
Let's outline the steps of the pipeline:
- Continuous integration: Push new code to GitHub, which is then built by CircleCI.
- Continuous deployment: Automatically deploy the new application to Heroku.
- Continuous testing: Execute post-deployment tests with Assertible.
I've created an example Node.js app for the code in this post. In the end, you'll have a workflow that will automatically build, deploy, and test your app after every push.
Continuous integration with CircleCI
The pipeline is initiated when code is pushed to a repository on GitHub. CircleCI will automatically pull the new changes, build the code, and run the unit tests. Setting this up only takes two steps:
- Add a
circle.ymlto our repository. You can see the example app's config file here.
- Tell CircleCI to watch the project. This is done in your CircleCI dashboard, by adding projects
Now, every time code is pushed to the GitHub repository, CircleCI will run the unit tests and fail if there are any issues:
Continuous deployment with Heroku
After our code has been built and the unit tests have passed, CircleCI
will look for a
deployment step in our configuration. In the example
app, I'm deploying the application to Heroku.
In the example project's
circle.yml file, take a look at the
deployment step again:
deployment: production: branch: master commands: # ...removed lines for brevity # Deploy the app to heroku - git push firstname.lastname@example.org:assertible-nodejs-example.git $CIRCLE_SHA1:refs/heads/master
This configuration tells CircleCI to deploy our application to
production when you push to the
master branch of the repo. The
deployment is executed by running each step in
This is continuous deployment: every time changes are pushed to the repo, CircleCI will build the app, and the deploy it to Heroku. Automatically. Every time. That's pretty cool!
Continuous testing with Assertible
After the application is deployed, you want to run an automated test suite on the new version to verify the release ensure it meets a certain standard. To do this, you use Assertible.
Let's look at the last couple of lines in our
deploying to Heroku:
# Download github_deploy script - curl https://raw.githubusercontent.com/assertible/deployments/master/github_deploy > github_deploy - chmod +x github_deploy # Send a deployment event to GitHub. GitHub will then send that # event to Assertible, and Assertible will run your API tests. - | APPROOT=https://assertible-nodejs-example.herokuapp.com/ DEPLOY_ID=$(ENVIRONMENT_NAME="production" ./github_deploy $CIRCLE_SHA1 $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME $APPROOT "pending") ./github_deploy $CIRCLE_SHA1 $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME $APPROOT "success" $DEPLOY_ID
github_deploy script to
send "deployment" events to our GitHub repository. Assertible watches
your repo for these events, and when a successful deployment occurs,
your API tests will automatically be executed. This is
post-deployment testing, a part of automated QA testing.
In the image below, you can see that Assertible tested the new deployment and ran the defined assertions on the application. It even says what commit the deployment was from:
Connecting Assertible to GitHub is easy! Just follow our GitHub Quick Start Guide.
Continuous testing is important for any team or developer releasing software. Automating API and QA tests not only saves time, but covers more cases faster than traditional, manual testing methods.
Testing staging environments from Pull Requests
A lot of times you're not deploying to production, but rather to a staging environment or Heroku review app. Assertible knows about your different environments and will run your API tests on the correct application.
When you deploy to a staging environment from a GitHub pull request, Assertible will show a status check that ensures all of your API tests are passing before the new changes are merged. Awesome!
Resources and examples
Even more automation and integrations
There's even more you can automate. Heroku has a ton of addons you can use to automate various parts of your build. Assertible supports integrations like Zapier that can be used to customize how you use your test results and get alerts of failures.
Integration with other CI services
Nothing about this pipeline is specific to GitHub, CircleCI, Heroku, or even Assertible. Learn more about setting up this pipeline with different CI providers, like Wercker, in the assertible/deployments repo.
Do you have anything to add to this list? Let me know!
What does your continuous development workflow look like? Shoot us a message on Twitter and let us know! Check it out and sign up for a free Assertible account to start testing every single deployment!
3 common deployment failures can be quickly detected and often prevented by automating end-to-end acceptance testing as part of the continuous integration process.
1. Database schema/model changes
Web applications and APIs that rely on a database inevitably change their schema or data layout. Changes to models and schema migrations can be extremely tricky and often have consequences which are difficult to predict, causing bugs, failures, and web service disruptions.
In an ideal world, failures related to schema migrations would be preventable. Sadly, bugs will still make their way into production in unpredictable ways. It's important to identify failure scenarios as fast as possible in production & non-production environments.
The ideal time to automatically run end-to-end tests against a web app is immediately after a deployment. This is especially true for teams who practice some form of continuous deployment; a wide surface-area of the application can be tested almost instantly after every deploy.
At Assertible, we test every single push to any branch of our code using the GitHub deployment integration. Using this integration, we can see the status of all our tests in each GitHub PR.
Every push to a branch deploys our web app and API to a staging
environment. When a branch is merged into
master, the code is built
and deployed to our
production environment, where we run a wide
range of tests against our web app and API.
If a deployment contains code that require model changes, we are able to execute our migrations against a staging database which mimics as much of our production systems as possible. When failures occur, we know instantly and can react accordingly.
By using this strategy, nearly all unexpected schema migration disruptions occur on a staging environment which allows us to adjust our patches accordingly.
In cases where performance, downtime, or data loss are big factors in maintaining backwards compatibility, we use branches dedicated to only the schema migration. We write our schema migrations in a backwards compatible way so follow-up branches can utilize the new models without disrupting the old models.
For teams that run migrations manually using SQL scripts, you can invoke a trigger url for your web service to run all the tests against your entire service immediately. Trigger URLs can be utilized from any point in your database maintenance process.
Make the smallest model changes possible
Use version control to record changes to the database
Have a rollback process in place and test it frequently
2. Service integration failures
Many web apps and APIs rely on external services to varying degrees. Services may be in the form of external APIs like the GitHub or Stripe, micro-services, or webhooks. These are all dependencies that reflect a point of failure in a modern web application architecture.
At Assertible, we solve integration failures using:
Unit and acceptance tests
We have isolated unit and acceptance tests which run during our continuous integration build phase. These tests use mocks for HTTP services and other outgoing requests. We make extensive use of mocking HTTP requests in our CI tests to represent external services.
We also have various acceptance tests which submit mocked versions of webhooks to handlers, ensuring they respond properly. This is all done during the build & test portion of our continuous integration pipeline.
Post-deployment / end-to-end tests
Part of our post-deployment testing process includes tests specifically designed for API endpoints and features which rely on external service integrations and webhooks. We use Slack alerts to immediately notify our team when anything goes wrong.
As a final protection against unknown failures, we try to log all corner-cases, exceptions, and unexpected error events (using Rollbar). This lets us know immediately when an unexpected error is hit and the corresponding metadata allows us to reproduce the test using Assertible.
3. Insufficient manual testing
Manual testing is generally done incrementally throughout the lifecycle of an app. On many teams, developers manually test each new feature as it's deployed. Larger enterprises may have a QA team with a more specialized set of tools but the process is still the same. As new features are built, only the newest and most critical features are tested, leaving holes in test coverage.
Problems with manual testing
- Time consuming
- Error prone
- Difficult to measure
The process of manually testing a web app each time it's deployed works well for single features which are new, but doesn't scale well for non-trivial apps that change frequently.
For each feature and bug our team develops, we write a new test using Assertible. We execute these tests every single time we deploy our web app and API to a staging or production environment. These tests then continue running on a schedule.
This has significantly reduced the burden of testing an application with a larger surface area and growing number of features and dependencies.
Automated testing best practices
Here's a more specific breakdown of how we use Assertible to test and identify various deployment failure scenarios:
Run tests after each and every deployment
Specifically, we use the Assertible GitHub deployment integration As mentioned above, we deploy all branches to a staging environment (as pull requests made through GitHub). Doing this maintains a robust continuous deployment pipeline where all pushes have several layers of testing; from unit tests to end-to-end tests.
Run tests on a schedule
Along with running tests on every deployment, we also run tests frequently on schedules. In the future we plan to expand the capabilities to run different types of tests at different frequencies. For example, tests that create and delete data should be run less often on production systems.
Write tests for new features
We create a new Assertible test for all new changes to our web app or API. A variety of workflows are tested and we're working towards having 100% API coverage.
We do NOT consider end-to-end tests a replacement for testing before deployment. We believe testing immediately after building and before deploying is crucial to keeping software high-quality and should be the first line of defense against bugs and regressions, followed by other kinds of integration tests and Assertible tests.
Write a test for each new regression or bug
We create new Assertible tests when we find bugs and regressions in our own code base. This helps us make sure they don't pop up again. We keep a record of all links to our issue tracker and pull requests in a test's description to conveniently lookup extended information about a specific test.
:: Christopher Reichert
API failures are inevitable; the important part is how quickly you can react. Here are 4 common API failures, and how you can test them.
Preventing and reacting to problems in your web services is something every developer, QA analyst, and project manager should be thinking about often. As web services grow they require more complex types of testing, but there are a few fundamentals that should always be covered in your automated API tests.
Here's how you can continuously monitor your web service for these issues using Assertible.
Invalid SSL certificates
It happens to the best of us: once a year an SSL cert expires and error reports start rolling in from API clients. These types of outages can be tough to track down and correct, so it's important to monitor your APIs SSL certificates and be alerted of any failures.
SSL validation testing with Assertible
a core assertion to validate SSL every time a
test is run. If the validation of your certificate doesn't pass, the
AssertValidSsl core assertion will fail the test. This
is configurable per environment, so if your
production web service requires SSL, but your
staging app doesn't,
you can configure your tests to only validate SSL in
Service provider outages
Sometimes it's not our fault. Service providers go out too. A web service becoming unreachable can cause widespread issues. Monitoring your API with a ping can help spot these situations fast.
Frequent health checks with Assertible
A basic request to test for
200 status codes is a good way to ensure
your service is reachable. You can add these tests to
a schedule on your web
service that runs on a given interval and alerts on any failures.
Too many redirects
Redirects in APIs and web applications are a part of how some services
work, but if something spins out of control you don't want Chrome
showing your users a
TOO_MANY_REDIRECTS page instead of your app.
Enforce a redirect limit
The Max redirects setting on each test sets a limit of the number of redirects the test's request can make. If it exceeds that number, the test will fail. Any API or web app that utilizes redirects, should have these cases tested.
Invalid payload formats
Sometimes applications return data in unexpected formats, like when a JSON API responds with an HTML error page, and things start to go wrong. It's important to validate your APIs response data, in normal and error scenarios, so your clients don't start to fail.
Payload validation with Assertible
Assertible has a JSON validation and an HTML validation assertion to test the response of your API. Every time your test is run, the response body will be checked and you will (hopefully!) see a passing assertion:
Having your bases covered with these steps is good start to correctly testing your APIs and web services. You can join Assertible for free and get started today!
Feedback or Questions? Tell us your thoughts on Twitter.
TL;DR Test APIs behind a firewall or on development machines using
1. download ngrok
The first step is to download a program to expose
another internal url) to the Internet. We'll use ngrok which is a
program can be downloaded for free and
available for all major operating systems.
Download and unpack ngrok.
2. start ngrok
The next step is to expose your web service. If your web service is on
localhost, simply run the following on the command-line:
$ ngrok http 3000
If you need to point
ngrok to another host behind a firewall
(e.g. inside an intranet), use the full host instead of a port only:
$ ngrok http internal.host:8099
You should see a line that looks something like this:
Forwarding http://7e9ea9dc.ngrok.io -> 127.0.0.1:3000
Forwarding http://7e9ea9dc.ngrok.io -> internal.host:8099
*.ngrok.io URL to your clipboard or write it down.
3. create a web service in Assertible
Next, create a web service in the Assertible dashboard.
Use the ngrok URL obtained in step 2 as the host of your Assertible
web service (e.g.
4. start testing!
Assertible will automatically run a test once when you create a new API to help you get oriented with the results.
You should see a similar output in
ngrok (although you may not have
From this point, you are completely setup and free to configure Assertible to test various aspects of your API or web app. Have fun!
Resources and documentation
:: Christopher Reichert
Instantly view GitHub pull request status checks for your post-deployment tests
We are very happy to release GitHub status checks for our post-deployment testing integration. Using Assertible alongside GitHub deployments, you can test and verify that your staging or production web services are operating correctly with end-to-end tests.
Resources and documentation
:: Christopher Reichert
TL;DR: Continuous development workflows make developers more productive by providing constant feedback during the development cycle. These 5 services will improve your CI pipeline.
There are a few stages to a good continuous development pipeline:
- Integration: Build code and run unit and integration tests
- Delivery: Deploy ('deliver') your new application
- Testing: Automated testing of staging and production environments
This post will go over what happens in every step of a good continuous workflow. Most of the tools and processes mentioned below are geared towards hosted services, particularly ones that integrate with a GitHub deployment pipeline.
Continuous integration is a process for frequently building and
testing new code changes. In most cases, a central repository (eg,
git) will have a
master branch where all changes are merged and
picked up by the CI build system.
With most tools, you will have a configuration file (eg,
.travis.yml) describing how to build the application and run unit
tests. As you push changes to a repository, your CI will build the
code from your config.
A good CI pipeline will ensure that all code can be built and run and that all unit tests are passing before being merged into the main branch for deployment.
There are many great hosted CI services that work with your existing GitHub repositories. It may take trying a few to see what works best for you. Two of the most widely used services are:
Free and paid plans available. Website
The modern continuous integration and delivery platform that software teams love to use. - CircleCI
Free and paid plans available. Website
Easily sync your GitHub projects with Travis CI and you’ll be testing your code in minutes! - Travis-CI
Continuous delivery is the process of deploying new versions of an application frequently, sometimes several times daily, to staging and production environments. An important part of continuous delivery is that it makes application changes more reliable by by providing a consistent, reproducable pipeline for pushing new changes.
Deploying an application into a staging or production environment can get hairy, but a good delivery system will have the automation to make it simple. Being confident about rapidly pushing code and having a good deployment process can greatly improve a product.
Like CI, there are a lot different tools for continuous delivery. One of the most powerful is Heroku Flow:
Free and paid plans available. Website
Heroku Flow brings together Heroku Pipelines, Review Apps and GitHub Integration into an easy to use structured workflow for continuous delivery, so you can test early and deploy often, with better results for your users. - Heroku
If you're team is already using GitHub to host code, the Heroku GitHub integration provides tools for setting up staging environments on pull requests ('Review Apps') and then deploying those to production.
The next step in a solid continuous development pipeline is testing. Usually, new code will be deployed to a staging location where changes can be tested and approved before reaching production. At this point, automated tests are run against the web service to ensure all services are working correctly.
A lot of times these same tests will be run against the production
environment as well. A dedicated staging or review area is not
required, but it does provide huge benefits. Another common work-flow
is that all changes merged into
master will be immediately deployed
to production. In these cases, your continuous testing should run as
soon as a new application is deployed.
Tools for continuous testing can range from manual QA teams and custom scripts for monitoring or testing API endpoints. Assertible is one hosted service that can run post-deployment from your CI pipeline.
Free and paid plans available. Sign up for a free account
Teams and individuals use Assertible to test and monitor production web services, create scheduled assertions, and gain confidence in crucial infrastructure. - Assertible
You are on the Assertible Blog so I might be a bit biased. Post deployment tests are, however, a crucial part of a good continuous development workflow that will enable you and your team to write more, better code.
What tools does your continuous development pipeline consist of? Shoot us a message on Twitter and let us know!
The assertible/deployments contains more information on integrating continuous testing into your existing workflow. Check it out and sign up for a free Assertible account to start testing your deployments!
TL;DR: Run tests against your web service every time you deploy a new version of your application from Heroku. Try it free.
In the Heroku dashboard, navigate to the app you would like to test and click the Deploy tab.
In the Deployment method section, select GitHub
In the Connect to GitHub section, select a repository (you may need to click Search, see the image below).
Once heroku recognizes your repo, click Connect. This should enable new options titled Automatic Deploys & Manual Deploys. The heroku dashboard should indicate that GitHub is now connected. For more information about setting up Heroku w/ GitHub, click here.
If you don't already have an Assertible account, you can get started by clicking the link below to sign in with GitHub.
Once you create an account, create your first Assertible test by entering the URL of a web service or importing a Swagger spec. Once you have created a test, navigate to the Deployments tab. Click the Add to GitHub button found on the GitHub integration and select a repository.
At this point, Assertible is completely configured to receive events from GitHub when your app is deployed. To test the integration, you can push code or do a manual deployment from the Heroku dashboard.
Resources and documentation
- Heroku has a seamless GitHub deployment integration
- GitHub repo with examples
- GitHub deployments integration docs
:: Christopher Reichert
Run tests against your web services every time you deploy a new version by connecting Assertible to your GitHub account. Try it free.
Assertible now connects with your GitHub account to run your test suites after deploying a new version of your application. Testing new features and deployments are a huge and crucial part of any developers workflow who runs web services. We aim to make this process better and easier, so we have a 3 step process for automatically testing new deployments:
How it works
Connect Assertible with GitHub
Assertible connects to your GitHub repository to receive deployment events. You can connect Assertible with your GitHub account by clicking on the button below:
Integrate with CI
Assertible tests will start running after a succesful deployment. A lot of continuous integration and deployment providers will work out of the box without any other configuration. For example, see how heroku deployments work with Assertible.
Example CI configurations
The assertible/deployments repository contains example configurations for popular CI services. Check out the repo to see how Assertible works with your CI provider.
Deploy code, run tests
Once you've got your GitHub account connected, and your deployment pipeline is set up to send events then you're all set! You can now start creating tests for your web service, making assertions, and setting up Slack notifications.
Resources and documentation
If you're ready to get started setting up your post deployment tests, check out these resources:
We're proud to announce Assertible, a tool for testing and monitoring production web services.
What is Assertible?
Assertible is a testing and monitoring service for websites and API's, primarily focusing on three goals:
- Easily create robust and meaningful assertions for your web app.
- Monitor your web services and notify you and your team when things aren't working as expected.
- Integrate with familiar third-party services (like Slack and GitHub!) so you can immediately reap the benefits, however you like to work.
The easiest way to get started is to import a Swagger spec (Swagger spec). The video below shows a quick demo setting up tests for the Hacker News API using a Swagger spec.
Sign up for a free account and try it out! You can start adding your own websites, get started with some of the demo API's, or add tests for any other service.
We would love to hear any of your feedback, you can reach us on the contact page.