Blog / engineering

Creating a setup step: Using variables in your tests

12/28/2016EngineeringCody Reichert

Using setup steps to capture variables and parameterize HTTP requests allows you to test a wide range of use-cases for your web service. Here's how it works with Assertible.

Signup for a free account to start testing your API

When testing an API, there are often cases where you need to use data from another HTTP request in your test. For example, to test a /users/{{userid}} endpoint, you first need to call /users to get a {{userid}}.

Assertible's setup steps give you the ability to do this by capturing variables and substituting them in your test before it is run. Let's walk through creating a test for our /users/{{userid}} endpoint and using a setup step to populate the {{userid}}.

Outline of our process

  1. Create a new test for the /users/{{userid}} endpoint.

  2. Add a setup step that populates the {{userid}} variable.

  3. Use our variable to make assertions on the HTTP response.

If you haven't set up your web service yet, you can get started with that here.

When we're done, we'll have a test for a dynamic API endpoint with assertions using dynamic data:

Testing a JSON API with route parameters

Create a new test for your web service

First let's create new API test. On your web service's overview page, find the "Manage" dropdown and choose "Create test" (docs). In the new test's request configuration, set the endpoint as /users/{{userid}}:

Create an API test with Assertible

The double-brace syntax around the {{userid}} in the endpoint is a test variable. Before the test is run, the variable will be populated with the data from our setup step.

If we run our test now, it will fail with a 404 because the {{userid}} variable wasn't populated with any data. Let's move to the next step and get our test using the correct user id.

Make a setup step and populate a variable

Now that we have a test for our API using a variable, we need to populate our test variable with some data. There are a couple ways to do this with Assertible:

  • HTTP request setup step (docs)
  • Random variable setup step (docs)
  • Hard-coded test variables (docs)

For our /users/{{userid}} endpoint we'll use the first option. Our setup step will make an HTTP request, before our test is run, to call the API's /users endpoint and save an id from the response to the {{userid}} variable.

Create a setup step by visiting the "Setups" tab of your new test and choosing "Add setup step". Next, configure the setup step to make an API request and capture a variable from the response. Your settings should look like this:

  • Name: Fetch userid variable
  • Generator: Select the HTTP request generator
  • Request settings: In this area, fill out the URL and other settings to call the /users endpoint.
  • Variable capture: Add one variable capture where:
    • Source: Select Body
    • Name: Our variable name is userid
    • Selector: In the /users JSON response, we want to select the first users ID: .[0].login

Assert API test setup step - variable capture

Learn more about creating setup steps here.

When you have everything filled out, save the setup step and enable it for this test. Now the {{userid}} variable in our API endpoint will be replaced with data from the setup step! If we run the test now, our 200 OK status code assertion will succeed and the test will be passing.

Add JSON response assertions using variables

Prop tip! Variables can be used in assertions as well as test settings like an endpoint. Learn more.

With a passing web service test, we can start creating better assertions for the API response and make our test more robust. Using the JSON path data assertion, we will test that the /users/{{userid}} endpoint is returning the correct data for the userid parameter.

On the "Assertions" tab of your test page click "Add assertion", and for the type of assertion choose JSON path data. This assertion allows us to use the JSON path selector syntax to assert that the .id field in the JSON API response is equal to our {{userid}} variable used in the endpoint:

JSON API response assertion

The Assertion Comparator will stay as Equals and the the Assertion Target will be the {{userid}} variable. If you save the assertion and run the test now, you'll see that our assertion target is populated with the {{userid}} value.

Every time our API test is run, we are asserting that the /users/{{userid}} route returns the correct user data for the user id that was provided. The end result will look like this:

Assertible API test result with variable capture

Wrapping up

To sum up what's happening in our test:

  • Before the test is run, a separate HTTP request is made to populate the {{userid}} variable.
  • Any usage of {{userid}} in our test will be replaced with the captured data.
  • The test is run with the replaced variables in our endpoint.
  • Assertions are made on the JSON API response.

There's a lot more you can with setup steps, test variables, and JSON path data assertions. Check out more examples in the docs and create your free Assertible account to get started today!

If you have any questions or feedback, send us a message or reach out on Twitter!

Assuredly yours,

Cody Reichert
Co-founder, Assertible

Full Article

Why we test every single API deployment

10/25/2016EngineeringCody Reichert

At Assertible, we automatically test every API deployment. Here's why.

Having a reliable process in place for testing and validating API deployments provides a huge amount of confidence when deploying new changes into production. It's the best way to know that your application is ready for your users, and it saves developers time by decreasing the need for manual testing.

In this post I'll discuss our continuous testing process, and how we use Assertible to validate releases with post-deployment testing.

Assertible GitHub Status Check

How we continuously test

At Assertible, our continuous integration/delivery/testing setup is pretty standard:

Running API tests automatically after every deployment quickly validates and provides feedback about the release.

For example, if a pull request is opened, the code will be built and deployed to staging. API tests are automatically run against the new app on the staging environment.

If this new deployment doesn't pass the Assertible tests, the GitHub status check on the PR will fail and tell us the code shouldn't be merged.

Assertible GitHub status check failing

More on the Assertible GitHub integration

By using Github we're able to take advantage of the Assertible/GitHub integration and deployment events.

After the code is built and in the process of being deployed to staging, Assertible receives deployment status events. When the deployment is successful, Assertible runs all tests for that API and updates the status check.

The assertible/deployments repo has instructions on setting up GitHub deployments as a part of your CI with a simple script.

Another neat part of using GitHub status checks with Assertible is that we can see the result of our API tests on the master branch. Any commit made to the master branch means a deployment to production. By looking in the commit history we can see that the API tests were all successful.

Assertible GitHub commit status check

API testing with Assertible

All API tests cases live in Assertible. There are tests set up to validate all API endpoints, test core functionality, and make assertions on response data integrity.

These are some of the tests that are run when a new version of the API is deployed.

Assertible API Testing

Before continuous testing, these all had to be run manually. The manual process is obviously time-consuming and error prone -- no one wants to do that.

With a solid continuous testing workflow in place, manually testing these endpoints is unnecessary. I can sleep sound knowing that everything has already been validated and changes can be pushed to production with confidence.

If a test fails, the status check on GitHub will have a link directly back to the failing test result to easily see why:

Assertible failing test

Uh oh! It looks like we broke some authentication logic in the /tests handler. This is where continuous testing saves our skin and our time (and in some cases our job!).

This is why we test every. single. deployment.

If all of the API tests are passing, we can happily merge knowing that the release has been validated and tested.

Side note! Assertible is free. If you're interested in setting this up yourself, you can get started here.

Continue the discussion

I always enjoy hearing how other teams test and validate API deployments. Tweet at me (@CodyReichert) or us (@AssertibleApp) and let's see what you're working with!

Assuredly yours,

Cody Reichert

Full Article

NPM Package: Lambda CloudWatch to Slack

07/26/2016EngineeringCody Reichert

This package provides a function to be run on AWS Lambda that sends CloudWatch notifications to your Slack channel. Check out the repo

Better Slack notifications for AWS CloudWatch

We released a new package for sending better AWS CloudWatch notifications to your Slack channel: lambda-cloudwatch-slack. Check it out on GitHub or NPM.

The repo provides a function that can be deployed to AWS Lambda and sends your CloudWatch alarms to a Slack channel. The README contains all the info you need on setting it up.

Improving default AWS messages

The default message aren't pretty and are tough to decipher. Using this function to send your messages improves on the default in several areas. Here's some screenshots:

AWS CloudWatch Notification for Slack

AWS CloudWatch Notification for Slack

Elastic Beanstalk notifications

Elastic Beanstalk Slack Notifications

Further improvements and contributing

If you have any feedback or ideas, feel free to open an issue on GitHub or send us a message on Twitter. Contributions are appreciated, you can open a pull request.

Assuredly yours,

Cody Reichert

Full Article

Creating email templates with React components

07/18/2016EngineeringCody Reichert

TL;DR - This post discusses and provides code samples for how we use React components to create our email templates.

The beauty of React.js is in the reusability and composition of components. Components are encapsualted pieces of UI that can be composed to build larger interfaces. At Assertible we use React.js for pretty much everything that you see: website pages, the entire dashboard, our email templates, and even this blog!

Emails are tough, so this post hopes to give a few pointers on how you can apply style attributes (for example, <p style={{...}}>) as well as classes (eg, <p className="..") to your React components, which allows for consitent rendering across email clients. Note that a lot of the information below is specific to webpack and babel.

The problem with emails

If you've ever written email templates, you know the pain of trying to get a consistent output in different email clients. Some will recognize <style> tags, while others will strip them; some require inline styles while others we still have no idea what they want.

This post provides an approach for using React.js components to create a module that be rendered to static markup (renderToStaticMarkup) and used in emails while keeping a consistent render output across clients. I won't cover all the caveats of email client's here, but if you're interested MailChimp has a good overview on the topic.

The goal output

In order to have a good output across email clients, we want to have our final markup contain both style attributes and class attributes. The class attributes are optional, since all the styles are inline as well, but I've found that it prevents some corner cases and it doesn't add too much bloat.


Now that we know we need both style attributes and classNames, we can come up with a goal before and after output for our markup.

This can be improved, of course, but ideally we can use the JSX spread operator to apply everything we need to an element at once. Something like this will work:


<p {['.text-muted', '.pull-xs-right']}>

The style function will take a list of 'classes' (classes that should be found within a CSS file -- more on that below), and apply both the styles and classes to our element:


<p class="text-muted pull-xs-right" style="color: gray; float: right">

With this approach we can guarantee a consistent render output across email clients. Gmail will be happy with the inline styles, and all the other clients will use whichever they prefer, classes or inline style attributes.

Let's go over the style function and what's happening:

style( Array<string> ): Object

The style function provides most of the functionality for applying the styles. Here is the entire code for style module (comments are important):

const bootstrap = require('/path/to/bootstrap.css')

export const applyStyle = (selectors, style = {}) => {
    // Our 'className' will be appended here
    let className = ""

    // For each 'selector' provided (eg '.pull-xs-right'), look up
    // the styles in the stylesheet. 'inline-style-loader', described
    // more below, is how we query the stylesheet.
    for(const key in selectors) {
        const slctr = selectors[key]
        const rules = bootstrap[slctr]
        // If CSS rules exist for this selector, add them to
        // 'className' and 'style'.
        if(rules) {
            const styles = rules.split(";")
            for (const key_ in styles) {
                const [attribute, value] = styles[key_].split(":")
                const attr = attribute.replace(/-([a-z])/g, g =>
                // add key/value to the 'style' object
                style[attr.trim()] = value.trim()

        // Append selector to className
        className += ` ${slctr.substr(1)}`

    // Return an object that can be spread directly onto a component.
    return {

If the comments don't provide enough info, the basic gist is: style takes an array of class names and returns an object with style and classNames ({ style, className }) that can be applied directly to a React component. The class names arguments are looked up in a style sheet (bootstrap in the example above) that is imported with inline-style-loader (discussed more below).

We now have a function that does this:

const styles = style(['.pull-xs-right'])
// {
//   className: "float:right;",
//   style: { float: 'right' },
// }


If you're wondering how we're able to query the CSS file as a JSON object to get access to styles, that is due to inline-style-loader. This webpack loader allows you to import a CSS file and access it as a JSON object.

One problem is that inline-style-loader will also format the styles to be used in a style _attribute (style="..."), so in the style function we have to to some string splitting.

inline-style-loader can be added to your webpack config like this:

module.exports = {
    // ... the rest of your config
    module: {
        loaders: [
            // ... the rest of your loaders
                test: /bootstrap.css$/,
                loader: "raw!inline-style"

Wrapping up

Using this method has been successful for us so far. We use this for all of our emails sent from the dashboard as of now. Here's an example signup email rendered in Gmail:

Assertible Getting Started Email

Try it out

Teams and individuals use Assertible to test and monitor production web services, create scheduled assertions, and gain confidence in crucial infrastructure. Sign up for a free acount today.

Sign in with GitHub

Assuredly yours,
Cody Reichert

Full Article

Simplified Assertions for Your Web Services

06/26/2016EngineeringCody Reichert

Testing your API shouldn't be harder than writing it.

Today we're happy to release new assertions for testing your website and API. Testing can quickly become complicated, but it shouldn't be. Assertible's simplified assertions allow you run known-to-be-valuable tests against your live web service. We've split the new assertions up into a couple of groups: Website 101 and API 101. To try these out on your own website, sign up for a free account and enter a URL.

Assertible - Simplified website testing

New Assertions

All of these links go to the documentation

Website 101

API 101

Simplifying assertions and how they're created makes your tests, and your time, more valuable. All of the new assertions can be enabled with just a click and some can take additional configuration options to make more specific assertions.

After running your test, you'll see the results of each assertion on the Test Results tab:

Assertible - Web service testing results

Create a free account and try it out!

Assuredly yours,

Cody Reichert

Full Article