Blog / featured


Tutorial: Set up a continuous testing pipeline with Node.js

03/22/2017FeaturedCody Reichert

This post will walk through setting up a continuous integration, deployment, and testing pipeline for a Node.js app using CircleCI, Heroku, and Assertible.

Continuous integration is large part of modern development workflows. Most of us are familiar with basic CI services that automatically build code as new changes are pushed. But what if you took that a step further and not only built your code, but also deployed and tested it within the same pipeline? This is continuous development.


Continuous development pipeline

Let's outline the steps of the pipeline:

I've created an example Node.js app for the code in this post. In the end, you'll have a workflow that will automatically build, deploy, and test your app after every push.

Continuous integration with CircleCI

The pipeline is initiated when code is pushed to a repository on GitHub. CircleCI will automatically pull the new changes, build the code, and run the unit tests. Setting this up only takes two steps:

  1. Add a circle.yml to our repository. You can see the example app's config file here.
  2. Tell CircleCI to watch the project. This is done in your CircleCI dashboard, by adding projects

The .circle.yml file contains the commands used to build the code. Since the example app is written in Node.js, the default settings will work (npm install) and you don't need to do anything else.

Now, every time code is pushed to the GitHub repository, CircleCI will run the unit tests and fail if there are any issues:


CircleCI successful build

Continuous deployment with Heroku

After our code has been built and the unit tests have passed, CircleCI will look for a deployment step in our configuration. In the example app, I'm deploying the application to Heroku.

In the example project's circle.yml file, take a look at the deployment step again:

deployment:
  production:
    branch: master
    commands:
      # ...removed lines for brevity
      # Deploy the app to heroku
      - git push git@heroku.com:assertible-nodejs-example.git $CIRCLE_SHA1:refs/heads/master

This configuration tells CircleCI to deploy our application to production when you push to the master branch of the repo. The deployment is executed by running each step in commands.

This is continuous deployment: every time changes are pushed to the repo, CircleCI will build the app, and the deploy it to Heroku. Automatically. Every time. That's pretty cool!


Heroku deployment activity

For more on deploying to Heroku, check out the Continuous deployment with Heroku docs on CircleCI or Continuously testing Heroku GitHub deployments blog on Assertible.

Continuous testing with Assertible

After the application is deployed, you want to run an automated test suite on the new version to verify the release ensure it meets a certain standard. To do this, you use Assertible.

Let's look at the last couple of lines in our circle.yml after deploying to Heroku:

# Download github_deploy script
- curl https://raw.githubusercontent.com/assertible/deployments/master/github_deploy > github_deploy
- chmod +x github_deploy

# Send a deployment event to GitHub. GitHub will then send that
# event to Assertible, and Assertible will run your API tests.
- |
  APPROOT=https://assertible-nodejs-example.herokuapp.com/
  DEPLOY_ID=$(ENVIRONMENT_NAME="production" ./github_deploy $CIRCLE_SHA1 $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME $APPROOT "pending")
  ./github_deploy $CIRCLE_SHA1 $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME $APPROOT "success" $DEPLOY_ID

This code uses the github_deploy script to send "deployment" events to our GitHub repository. Assertible watches your repo for these events, and when a successful deployment occurs, your API tests will automatically be executed. This is post-deployment testing, a part of automated QA testing.

In the image below, you can see that Assertible tested the new deployment and ran the defined assertions on the application. It even says what commit the deployment was from:


Post-deployment API testing result

Connecting Assertible to GitHub is easy! Just follow our GitHub Quick Start Guide.

Continuous testing is important for any team or developer releasing software. Automating API and QA tests not only saves time, but covers more cases faster than traditional, manual testing methods.


Start building your CI pipeline
Assertible is free to use. Contact us if you have any questions or feedback!


Testing staging environments from Pull Requests

A lot of times you're not deploying to production, but rather to a staging environment or Heroku review app. Assertible knows about your different environments and will run your API tests on the correct application.

When you deploy to a staging environment from a GitHub pull request, Assertible will show a status check that ensures all of your API tests are passing before the new changes are merged. Awesome!


GitHub status checks for continuous testing

Resources and examples

Even more automation and integrations

There's even more you can automate. Heroku has a ton of addons you can use to automate various parts of your build. Assertible supports integrations like Zapier that can be used to customize how you use your test results and get alerts of failures.

Integration with other CI services

Nothing about this pipeline is specific to GitHub, CircleCI, Heroku, or even Assertible. Learn more about setting up this pipeline with different CI providers, like Wercker, in the assertible/deployments repo.

Do you have anything to add to this list? Let me know!

What does your continuous development workflow look like? Shoot us a message on Twitter and let us know! Check it out and sign up for a free Assertible account to start testing every single deployment!


Assuredly yours,
Cody Reichert

Full Article


3 preventable web app deployment failures

03/11/2017FeaturedChristopher Reichert

TL;DR:

3 common deployment failures can be quickly detected and often prevented by automating end-to-end acceptance testing as part of the continuous integration process.

1. Database schema/model changes

Web applications and APIs that rely on a database inevitably change their schema or data layout. Changes to models and schema migrations can be extremely tricky and often have consequences which are difficult to predict, causing bugs, failures, and web service disruptions.

In an ideal world, failures related to schema migrations would be preventable. Sadly, bugs will still make their way into production in unpredictable ways. It's important to identify failure scenarios as fast as possible in production & non-production environments.

The ideal time to automatically run end-to-end tests against a web app is immediately after a deployment. This is especially true for teams who practice some form of continuous deployment; a wide surface-area of the application can be tested almost instantly after every deploy.

At Assertible, we test every single push to any branch of our code using the GitHub deployment integration. Using this integration, we can see the status of all our tests in each GitHub PR.

Assertible GitHub deployment integration

Every push to a branch deploys our web app and API to a staging environment. When a branch is merged into master, the code is built and deployed to our production environment, where we run a wide range of tests against our web app and API.

If a deployment contains code that require model changes, we are able to execute our migrations against a staging database which mimics as much of our production systems as possible. When failures occur, we know instantly and can react accordingly.

By using this strategy, nearly all unexpected schema migration disruptions occur on a staging environment which allows us to adjust our patches accordingly.

In cases where performance, downtime, or data loss are big factors in maintaining backwards compatibility, we use branches dedicated to only the schema migration. We write our schema migrations in a backwards compatible way so follow-up branches can utilize the new models without disrupting the old models.


For teams that run migrations manually using SQL scripts, you can invoke a trigger url for your web service to run all the tests against your entire service immediately. Trigger URLs can be utilized from any point in your database maintenance process.

In all cases, you can receive failure notifications to Slack and email or check the Assertible dashboard


Deployment tips

  • Make the smallest model changes possible

  • Use version control to record changes to the database

  • Have a rollback process in place and test it frequently


2. Service integration failures

Many web apps and APIs rely on external services to varying degrees. Services may be in the form of external APIs like the GitHub or Stripe, micro-services, or webhooks. These are all dependencies that reflect a point of failure in a modern web application architecture.

At Assertible, we solve integration failures using:

  • Unit and acceptance tests

    We have isolated unit and acceptance tests which run during our continuous integration build phase. These tests use mocks for HTTP services and other outgoing requests. We make extensive use of mocking HTTP requests in our CI tests to represent external services.

    We also have various acceptance tests which submit mocked versions of webhooks to handlers, ensuring they respond properly. This is all done during the build & test portion of our continuous integration pipeline.

  • Post-deployment / end-to-end tests

    Part of our post-deployment testing process includes tests specifically designed for API endpoints and features which rely on external service integrations and webhooks. We use Slack alerts to immediately notify our team when anything goes wrong.

  • Event logging

    As a final protection against unknown failures, we try to log all corner-cases, exceptions, and unexpected error events (using Rollbar). This lets us know immediately when an unexpected error is hit and the corresponding metadata allows us to reproduce the test using Assertible.


3. Insufficient manual testing

Manual testing is generally done incrementally throughout the lifecycle of an app. On many teams, developers manually test each new feature as it's deployed. Larger enterprises may have a QA team with a more specialized set of tools but the process is still the same. As new features are built, only the newest and most critical features are tested, leaving holes in test coverage.

QA Testing Manually

Problems with manual testing

  • Time consuming
  • Error prone
  • Difficult to measure

The process of manually testing a web app each time it's deployed works well for single features which are new, but doesn't scale well for non-trivial apps that change frequently.

For each feature and bug our team develops, we write a new test using Assertible. We execute these tests every single time we deploy our web app and API to a staging or production environment. These tests then continue running on a schedule.

This has significantly reduced the burden of testing an application with a larger surface area and growing number of features and dependencies.


Automated testing best practices

Here's a more specific breakdown of how we use Assertible to test and identify various deployment failure scenarios:

  • Run tests after each and every deployment

    Specifically, we use the Assertible GitHub deployment integration As mentioned above, we deploy all branches to a staging environment (as pull requests made through GitHub). Doing this maintains a robust continuous deployment pipeline where all pushes have several layers of testing; from unit tests to end-to-end tests.

  • Run tests on a schedule

    Along with running tests on every deployment, we also run tests frequently on schedules. In the future we plan to expand the capabilities to run different types of tests at different frequencies. For example, tests that create and delete data should be run less often on production systems.

  • Write tests for new features

    We create a new Assertible test for all new changes to our web app or API. A variety of workflows are tested and we're working towards having 100% API coverage.

    We do NOT consider end-to-end tests a replacement for testing before deployment. We believe testing immediately after building and before deploying is crucial to keeping software high-quality and should be the first line of defense against bugs and regressions, followed by other kinds of integration tests and Assertible tests.

  • Write a test for each new regression or bug

    We create new Assertible tests when we find bugs and regressions in our own code base. This helps us make sure they don't pop up again. We keep a record of all links to our issue tracker and pull requests in a test's description to conveniently lookup extended information about a specific test.

:: Christopher Reichert

Full Article


4 common API errors and how to test them

02/10/2017FeaturedCody Reichert

API failures are inevitable; the important part is how quickly you can react. Here are 4 common API failures, and how you can test them.

Preventing and reacting to problems in your web services is something every developer, QA analyst, and project manager should be thinking about often. As web services grow they require more complex types of testing, but there are a few fundamentals that should always be covered in your automated API tests.

  1. Invalid SSL certificates
  2. Service provider outages
  3. Too many redirects
  4. Invalid payload formats

Here's how you can continuously monitor your web service for these issues using Assertible.

Try Assertible for free to start testing your API

Invalid SSL certificates

It happens to the best of us: once a year an SSL cert expires and error reports start rolling in from API clients. These types of outages can be tough to track down and correct, so it's important to monitor your APIs SSL certificates and be alerted of any failures.

SSL validation testing with Assertible

Assertible has a core assertion to validate SSL every time a test is run. If the validation of your certificate doesn't pass, the AssertValidSsl core assertion will fail the test. This is configurable per environment, so if your production web service requires SSL, but your staging app doesn't, you can configure your tests to only validate SSL in production.




Service provider outages

Sometimes it's not our fault. Service providers go out too. A web service becoming unreachable can cause widespread issues. Monitoring your API with a ping can help spot these situations fast.

Frequent health checks with Assertible

A basic request to test for 200 status codes is a good way to ensure your service is reachable. You can add these tests to a schedule on your web service that runs on a given interval and alerts on any failures.


GitHub Web Service - Passing Assertible Tests

Too many redirects

Redirects in APIs and web applications are a part of how some services work, but if something spins out of control you don't want Chrome showing your users a TOO_MANY_REDIRECTS page instead of your app.

Enforce a redirect limit

The Max redirects setting on each test sets a limit of the number of redirects the test's request can make. If it exceeds that number, the test will fail. Any API or web app that utilizes redirects, should have these cases tested.


Configure max redirects for an API test

Invalid payload formats

Sometimes applications return data in unexpected formats, like when a JSON API responds with an HTML error page, and things start to go wrong. It's important to validate your APIs response data, in normal and error scenarios, so your clients don't start to fail.

Payload validation with Assertible

Assertible has a JSON validation and an HTML validation assertion to test the response of your API. Every time your test is run, the response body will be checked and you will (hopefully!) see a passing assertion:


JSON payload validation with Assertible


Having your bases covered with these steps is good start to correctly testing your APIs and web services. You can join Assertible for free and get started today!

Feedback or Questions? Tell us your thoughts on Twitter.

Assuredly yours,
Cody Reichert
Co-founder Assertible

Full Article


Testing an API behind a firewall

02/08/2017FeaturedChristopher Reichert

TL;DR Test APIs behind a firewall or on development machines using ngrok

Try it for yourself with a free Assertible account

1. download ngrok

The first step is to download a program to expose localhost (or another internal url) to the Internet. We'll use ngrok which is a program can be downloaded for free and available for all major operating systems.

https://ngrok.com/download

Download and unpack ngrok.

2. start ngrok

The next step is to expose your web service. If your web service is on localhost, simply run the following on the command-line:

$ ngrok http 3000

If you need to point ngrok to another host behind a firewall (e.g. inside an intranet), use the full host instead of a port only:

$ ngrok http internal.host:8099

You should see a line that looks something like this:

Forwarding    http://7e9ea9dc.ngrok.io -> 127.0.0.1:3000

or

Forwarding    http://7e9ea9dc.ngrok.io -> internal.host:8099

Copy the *.ngrok.io URL to your clipboard or write it down.

3. create a web service in Assertible

Next, create a web service in the Assertible dashboard.

Create an API pointed at an ngrok url

Use the ngrok URL obtained in step 2 as the host of your Assertible web service (e.g. http://7e9ea9dc.ngrok.io).

4. start testing!

Assertible will automatically run a test once when you create a new API to help you get oriented with the results.

ngrok Assertible test result

You should see a similar output in ngrok (although you may not have any failures!).

ngrok output

From this point, you are completely setup and free to configure Assertible to test various aspects of your API or web app. Have fun!

Resources and documentation


:: Christopher Reichert

Full Article


GitHub integration status checks

07/29/2016FeaturedChristopher Reichert

Instantly view GitHub pull request status checks for your post-deployment tests

Try the GitHub integration with a free account

We are very happy to release GitHub status checks for our post-deployment testing integration. Using Assertible alongside GitHub deployments, you can test and verify that your staging or production web services are operating correctly with end-to-end tests.

GitHub Status Checks with Assertible

To get started, sign in with GitHub and check out the open source scripts in our deployments repo.

Resources and documentation

:: Christopher Reichert

Full Article


5 tools to complete your CI pipeline

07/20/2016FeaturedCody Reichert

TL;DR: Continuous development workflows make developers more productive by providing constant feedback during the development cycle. These 5 services will improve your CI pipeline.

Continuous development

Continuous integration, delivery, and testing provide teams with a way to frequently build new code, deploy changes to applications, and run tests before and after each step.

There are a few stages to a good continuous development pipeline:

  1. Integration: Build code and run unit and integration tests
  2. Delivery: Deploy ('deliver') your new application
  3. Testing: Automated testing of staging and production environments

This post will go over what happens in every step of a good continuous workflow. Most of the tools and processes mentioned below are geared towards hosted services, particularly ones that integrate with a GitHub deployment pipeline.

Continuous integration

Continuous integration is a process for frequently building and testing new code changes. In most cases, a central repository (eg, git) will have a master branch where all changes are merged and picked up by the CI build system.

With most tools, you will have a configuration file (eg, .travis.yml) describing how to build the application and run unit tests. As you push changes to a repository, your CI will build the code from your config.

A good CI pipeline will ensure that all code can be built and run and that all unit tests are passing before being merged into the main branch for deployment.

Tools

There are many great hosted CI services that work with your existing GitHub repositories. It may take trying a few to see what works best for you. Two of the most widely used services are:

Circle CI

Free and paid plans available. Website

The modern continuous integration and delivery platform that software teams love to use. - CircleCI

Travis CI TravisCI Logo

Free and paid plans available. Website

Easily sync your GitHub projects with Travis CI and you’ll be testing your code in minutes! - Travis-CI

Continuous delivery

Continuous delivery is the process of deploying new versions of an application frequently, sometimes several times daily, to staging and production environments. An important part of continuous delivery is that it makes application changes more reliable by by providing a consistent, reproducable pipeline for pushing new changes.

Deploying an application into a staging or production environment can get hairy, but a good delivery system will have the automation to make it simple. Being confident about rapidly pushing code and having a good deployment process can greatly improve a product.

Tools

Like CI, there are a lot different tools for continuous delivery. One of the most powerful is Heroku Flow:

Heroku Flow Heroku logo

Free and paid plans available. Website

Heroku Flow brings together Heroku Pipelines, Review Apps and GitHub Integration into an easy to use structured workflow for continuous delivery, so you can test early and deploy often, with better results for your users. - Heroku

If you're team is already using GitHub to host code, the Heroku GitHub integration provides tools for setting up staging environments on pull requests ('Review Apps') and then deploying those to production.

Continuous testing

The next step in a solid continuous development pipeline is testing. Usually, new code will be deployed to a staging location where changes can be tested and approved before reaching production. At this point, automated tests are run against the web service to ensure all services are working correctly.

A lot of times these same tests will be run against the production environment as well. A dedicated staging or review area is not required, but it does provide huge benefits. Another common work-flow is that all changes merged into master will be immediately deployed to production. In these cases, your continuous testing should run as soon as a new application is deployed.

Tools

Tools for continuous testing can range from manual QA teams and custom scripts for monitoring or testing API endpoints. Assertible is one hosted service that can run post-deployment from your CI pipeline.

Assertible

Free and paid plans available. Sign up for a free account

Teams and individuals use Assertible to test and monitor production web services, create scheduled assertions, and gain confidence in crucial infrastructure. - Assertible

You are on the Assertible Blog so I might be a bit biased. Post deployment tests are, however, a crucial part of a good continuous development workflow that will enable you and your team to write more, better code.

Wrapping up

What tools does your continuous development pipeline consist of? Shoot us a message on Twitter and let us know!

The assertible/deployments contains more information on integrating continuous testing into your existing workflow. Check it out and sign up for a free Assertible account to start testing your deployments!


Assuredly yours,
Cody Reichert

Full Article


Continuously testing Heroku GitHub deployments

07/11/2016FeaturedChristopher Reichert

TL;DR: Run tests against your web service every time you deploy a new version of your application from Heroku. Try it free.

Heroku logo

Setup

  1. Configure GitHub to connect w/ Heroku

    In the Heroku dashboard, navigate to the app you would like to test and click the Deploy tab.

    Heroku dashboard

    In the Deployment method section, select GitHub

    In the Connect to GitHub section, select a repository (you may need to click Search, see the image below).

    Heroku choose GitHub repository

    Once heroku recognizes your repo, click Connect. This should enable new options titled Automatic Deploys & Manual Deploys. The heroku dashboard should indicate that GitHub is now connected. For more information about setting up Heroku w/ GitHub, click here.

  2. Connect Assertible w/ GitHub

    If you don't already have an Assertible account, you can get started by clicking the link below to sign in with GitHub.

    Sign in with GitHub

    Once you create an account, create your first Assertible test by entering the URL of a web service or importing a Swagger spec. Once you have created a test, navigate to the Deployments tab. Click the Add to GitHub button found on the GitHub integration and select a repository.

    Fig 1.0 GitHub Deployments Integration

  3. Deploy your web service

    At this point, Assertible is completely configured to receive events from GitHub when your app is deployed. To test the integration, you can push code or do a manual deployment from the Heroku dashboard.

Resources and documentation

:: Christopher Reichert

Full Article


Effortless post deployment testing with GitHub

07/10/2016FeaturedCody Reichert

Run tests against your web services every time you deploy a new version by connecting Assertible to your GitHub account. Try it free.

Assertible now connects with your GitHub account to run your test suites after deploying a new version of your application. Testing new features and deployments are a huge and crucial part of any developers workflow who runs web services. We aim to make this process better and easier, so we have a 3 step process for automatically testing new deployments:

How it works

Test deployments with Assertible


Connect Assertible with GitHub

Assertible connects to your GitHub repository to receive deployment events. You can connect Assertible with your GitHub account by clicking on the button below:

Sign in with GitHub

Integrate with CI

Assertible tests will start running after a succesful deployment. A lot of continuous integration and deployment providers will work out of the box without any other configuration. For example, see how heroku deployments work with Assertible.

Example CI configurations

The assertible/deployments repository contains example configurations for popular CI services. Check out the repo to see how Assertible works with your CI provider.

Deploy code, run tests

Once you've got your GitHub account connected, and your deployment pipeline is set up to send events then you're all set! You can now start creating tests for your web service, making assertions, and setting up Slack notifications.

Resources and documentation

If you're ready to get started setting up your post deployment tests, check out these resources:

Assuredly yours,

Cody Reichert

Full Article


Introducing Assertible

04/25/2016FeaturedCody Reichert

We're proud to announce Assertible, a tool for testing and monitoring production web services.

What is Assertible?

Assertible is a testing and monitoring service for websites and API's, primarily focusing on three goals:

  • Easily create robust and meaningful assertions for your web app.
  • Monitor your web services and notify you and your team when things aren't working as expected.
  • Integrate with familiar third-party services (like Slack and GitHub!) so you can immediately reap the benefits, however you like to work.

Getting started

The easiest way to get started is to import a Swagger spec (Swagger spec). The video below shows a quick demo setting up tests for the Hacker News API using a Swagger spec.


Learn more

Sign up for a free account and try it out! You can start adding your own websites, get started with some of the demo API's, or add tests for any other service.


We would love to hear any of your feedback, you can reach us on the contact page.

Assuredly yours,

Cody Reichert

Full Article