Blog


New feature: dashboard environment context

3/28/2017ChangelogCody Reichert

Execute your web service tests against any environment directly from the dashboard, and get a status badge for all of your environments. Start testing today.

Continuous testing and monitoring takes place throughout the entire development cycle: from writing features on your local machine, to testing changes on staging environments, and deploying new versions of your application to production. Your automated API tests should help you understand the status of each environment and how your tests are performing across releases.

Assertible has first-class support for testing different web service environments via deployments and Trigger URLs. Today, we're proud to release new features that allow you to execute tests on any of your environments directly from the dashboard!

Run tests against different web service environments

This gives teams a way to run pre-defined tests against any instance of your application - like a staging environment or review app. Using a tool like ngrok, you can even test a web service running on localhost, tightening the feedback loop for API developers writing new features or bug fixes.


Test your web services
Try Assertible for free. Have any questions? Contact us any time!


Environments for status badges

Web service status badges now support an environment parameter to instantly view the status of your APIs from anywhere. Status badges can be used anywhere, like in a GitHub README or API documentation, to give your users insight into the status of your services. Learn more about getting a status badge for your APIs in the docs.

Web service status badges for environments

Web service environments and status badges are available for all accounts, free and paid. Sign up today and start continuously testing your web services. Is there a feature for environments you'd like to see? Send us a message or reach out on Twitter and let's talk!

:: Cody Reichert

Full Article


New feature: web service monitoring metrics

3/27/2017ChangelogCody Reichert

Introducing new metrics for testing and monitoring your web services. Start monitoring today.

An essential part of testing your APIs and web services is having access to a real-time view of performance, problematic endpoints, and service accessibility. API status needs to be more than a pass or fail to really understand what your application's consumers are experiencing.

Assertible API testing and monitoring metrics

Assertible's new web service monitoring and metrics gives insight into the health of your services. Developers and QA teams can now quickly see average response times for requests, average pass rate for the service, and what failures occured in the last few hours. Every time your tests are run, the metrics for your service will be updated based on the most recent test results.


Start monitoring today
Use Assertible for free. Have any questions? Contact us any time!


The new web service monitoring metrics are available for all accounts and services in the Assertible dashboard. Is there a metric or data point you would like to see for your API? Let us know or reach out on Twitter, we'd love to talk testing!

:: Cody Reichert

Full Article


New feature: deployments API

3/27/2017ChangelogChristopher Reichert

Trigger post-deployment tests and track releases using the Assertible deployments API

The best time to execute automated API tests is immediately after a deployment. Ideally, tests are run as part of a continuous integration and testing pipeline.

Our new Deployments API is designed to make post-deployment testing a first class workflow in continuous integration pipelines. When a new deployment is posted in Assertible, all API tests are executed immediately.


Deployments API Diagram

There are two ways to utilize the deployments API:

  1. From the command-line:

     curl -u access_token: -XPOST 'https://assertible.com/deployments -d'{
         "service" : "00000000-0000-0000-0000-000000000000",
         "environmentName" : "staging",
         "version" : "v1"
     }'

    NOTE: This example uses sample data. See the deployment documentation for more examples.

  2. Integrations:

    For example, the GitHub deployments integration. In the picture below, a deployment is created and tests are executed against the assertible-staging environment:

    Assertible GitHub status check

    NOTE: Environments can be configured in the assertible dashboard. See the environment documentation for more examples.

Currently, GitHub is the only supported deployment integration. We plan on having more integrations with deployments like Heroku, Zapier, and Amazon AWS.


Start building your CI pipeline
Assertible is free to use. Contact us if you have any questions or feedback!


Automating API tests after a deployment alleviates the need for redundant manual testing and can also reduce error rates in web services. If you have any questions about continuous testing or feedback on our new API, shoot us a message.

Resources,

:: Christopher Reichert

Full Article


Tutorial: Set up a continuous testing pipeline with Node.js

03/22/2017FeaturedCody Reichert

This post will walk through setting up a continuous integration, deployment, and testing pipeline for a Node.js app using CircleCI, Heroku, and Assertible.

Continuous integration is large part of modern development workflows. Most of us are familiar with basic CI services that automatically build code as new changes are pushed. But what if you took that a step further and not only built your code, but also deployed and tested it within the same pipeline? This is continuous development.


Continuous development pipeline

Let's outline the steps of the pipeline:

I've created an example Node.js app for the code in this post. In the end, you'll have a workflow that will automatically build, deploy, and test your app after every push.

Continuous integration with CircleCI

The pipeline is initiated when code is pushed to a repository on GitHub. CircleCI will automatically pull the new changes, build the code, and run the unit tests. Setting this up only takes two steps:

  1. Add a circle.yml to our repository. You can see the example app's config file here.
  2. Tell CircleCI to watch the project. This is done in your CircleCI dashboard, by adding projects

The .circle.yml file contains the commands used to build the code. Since the example app is written in Node.js, the default settings will work (npm install) and you don't need to do anything else.

Now, every time code is pushed to the GitHub repository, CircleCI will run the unit tests and fail if there are any issues:


CircleCI successful build

Continuous deployment with Heroku

After our code has been built and the unit tests have passed, CircleCI will look for a deployment step in our configuration. In the example app, I'm deploying the application to Heroku.

In the example project's circle.yml file, take a look at the deployment step again:

deployment:
  production:
    branch: master
    commands:
      # ...removed lines for brevity
      # Deploy the app to heroku
      - git push git@heroku.com:assertible-nodejs-example.git $CIRCLE_SHA1:refs/heads/master

This configuration tells CircleCI to deploy our application to production when you push to the master branch of the repo. The deployment is executed by running each step in commands.

This is continuous deployment: every time changes are pushed to the repo, CircleCI will build the app, and the deploy it to Heroku. Automatically. Every time. That's pretty cool!


Heroku deployment activity

For more on deploying to Heroku, check out the Continuous deployment with Heroku docs on CircleCI or Continuously testing Heroku GitHub deployments blog on Assertible.

Continuous testing with Assertible

After the application is deployed, you want to run an automated test suite on the new version to verify the release ensure it meets a certain standard. To do this, you use Assertible.

Let's look at the last couple of lines in our circle.yml after deploying to Heroku:

# Download github_deploy script
- curl https://raw.githubusercontent.com/assertible/deployments/master/github_deploy > github_deploy
- chmod +x github_deploy

# Send a deployment event to GitHub. GitHub will then send that
# event to Assertible, and Assertible will run your API tests.
- |
  APPROOT=https://assertible-nodejs-example.herokuapp.com/
  DEPLOY_ID=$(ENVIRONMENT_NAME="production" ./github_deploy $CIRCLE_SHA1 $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME $APPROOT "pending")
  ./github_deploy $CIRCLE_SHA1 $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME $APPROOT "success" $DEPLOY_ID

This code uses the github_deploy script to send "deployment" events to our GitHub repository. Assertible watches your repo for these events, and when a successful deployment occurs, your API tests will automatically be executed. This is post-deployment testing, a part of automated QA testing.

In the image below, you can see that Assertible tested the new deployment and ran the defined assertions on the application. It even says what commit the deployment was from:


Post-deployment API testing result

Connecting Assertible to GitHub is easy! Just follow our GitHub Quick Start Guide.

Continuous testing is important for any team or developer releasing software. Automating API and QA tests not only saves time, but covers more cases faster than traditional, manual testing methods.


Start building your CI pipeline
Assertible is free to use. Contact us if you have any questions or feedback!


Testing staging environments from Pull Requests

A lot of times you're not deploying to production, but rather to a staging environment or Heroku review app. Assertible knows about your different environments and will run your API tests on the correct application.

When you deploy to a staging environment from a GitHub pull request, Assertible will show a status check that ensures all of your API tests are passing before the new changes are merged. Awesome!


GitHub status checks for continuous testing

Resources and examples

Even more automation and integrations

There's even more you can automate. Heroku has a ton of addons you can use to automate various parts of your build. Assertible supports integrations like Zapier that can be used to customize how you use your test results and get alerts of failures.

Integration with other CI services

Nothing about this pipeline is specific to GitHub, CircleCI, Heroku, or even Assertible. Learn more about setting up this pipeline with different CI providers, like Wercker, in the assertible/deployments repo.

Do you have anything to add to this list? Let me know!

What does your continuous development workflow look like? Shoot us a message on Twitter and let us know! Check it out and sign up for a free Assertible account to start testing every single deployment!


Assuredly yours,
Cody Reichert

Full Article


New feature: via for test results

3/21/2017ChangelogChristopher Reichert

Assertible's new via property for test results makes it trivial to instantly view where a test result originated.

When API tests fail, it's imperative to understand the source of the failure as quickly as possible. Assertible's new via property for test results allows teams and individuals to instantly view where test results originate. For example, whether a test was executed on a schedule, deployment, or in the dashboard by another teammate, all of which are important indicators on how to properly resolve an issue.

Start testing your web service today with a free account

Knowing how and where tests are being run gives you a better monitoring overview of your API. The test result via property is shown in various places across the dashboard, like in the "Latest activity" stream for a web service:

Assertible activity stream viewing test result via

The test result via is also displayed prominently in our Slack integration:

Assertible failing test result via seen in Slack

The quicker API and web app failures can be understood, the less time teams spend resolving failures.

If you have a feature request or questions, feel free to send us a message or reach out on Twitter any time.

:: Christopher Reichert

Full Article


New feature: Zapier Integration

3/20/2017ChangelogCody Reichert

We're excited to announce that Assertible has a new integration with Zapier; create a Zap that triggers when your API tests are run to send results to other services.

Try the Zapier integration with a free account

Zapier is a service that makes it easy to send data between web apps. Assertible's new Zapier integration allows you trigger a Zap when your API tests are run. The test result is then sent to Zapier, and can be used in other services; like logging results in a Google spreadsheet, or sending a text message when tests fail.


Using your test result data to augment other metrics and services opens up a wide range of possibilities for managing and testing your API. Learn more about the Zapier integration and how to start working with your web services in the the documentation.

All of Assertible's integrations are available on free accounts, so you can get started today. The Zapier integration is currently in beta, so if you have a feature request or questions, feel free to send us a message or reach out on Twitter any time.

:: Cody Reichert

Full Article


3 preventable web app deployment failures

03/11/2017FeaturedChristopher Reichert

TL;DR:

3 common deployment failures can be quickly detected and often prevented by automating end-to-end acceptance testing as part of the continuous integration process.

1. Database schema/model changes

Web applications and APIs that rely on a database inevitably change their schema or data layout. Changes to models and schema migrations can be extremely tricky and often have consequences which are difficult to predict, causing bugs, failures, and web service disruptions.

In an ideal world, failures related to schema migrations would be preventable. Sadly, bugs will still make their way into production in unpredictable ways. It's important to identify failure scenarios as fast as possible in production & non-production environments.

The ideal time to automatically run end-to-end tests against a web app is immediately after a deployment. This is especially true for teams who practice some form of continuous deployment; a wide surface-area of the application can be tested almost instantly after every deploy.

At Assertible, we test every single push to any branch of our code using the GitHub deployment integration. Using this integration, we can see the status of all our tests in each GitHub PR.

Assertible GitHub deployment integration

Every push to a branch deploys our web app and API to a staging environment. When a branch is merged into master, the code is built and deployed to our production environment, where we run a wide range of tests against our web app and API.

If a deployment contains code that require model changes, we are able to execute our migrations against a staging database which mimics as much of our production systems as possible. When failures occur, we know instantly and can react accordingly.

By using this strategy, nearly all unexpected schema migration disruptions occur on a staging environment which allows us to adjust our patches accordingly.

In cases where performance, downtime, or data loss are big factors in maintaining backwards compatibility, we use branches dedicated to only the schema migration. We write our schema migrations in a backwards compatible way so follow-up branches can utilize the new models without disrupting the old models.


For teams that run migrations manually using SQL scripts, you can invoke a trigger url for your web service to run all the tests against your entire service immediately. Trigger URLs can be utilized from any point in your database maintenance process.

In all cases, you can receive failure notifications to Slack and email or check the Assertible dashboard


Deployment tips

  • Make the smallest model changes possible

  • Use version control to record changes to the database

  • Have a rollback process in place and test it frequently


2. Service integration failures

Many web apps and APIs rely on external services to varying degrees. Services may be in the form of external APIs like the GitHub or Stripe, micro-services, or webhooks. These are all dependencies that reflect a point of failure in a modern web application architecture.

At Assertible, we solve integration failures using:

  • Unit and acceptance tests

    We have isolated unit and acceptance tests which run during our continuous integration build phase. These tests use mocks for HTTP services and other outgoing requests. We make extensive use of mocking HTTP requests in our CI tests to represent external services.

    We also have various acceptance tests which submit mocked versions of webhooks to handlers, ensuring they respond properly. This is all done during the build & test portion of our continuous integration pipeline.

  • Post-deployment / end-to-end tests

    Part of our post-deployment testing process includes tests specifically designed for API endpoints and features which rely on external service integrations and webhooks. We use Slack alerts to immediately notify our team when anything goes wrong.

  • Event logging

    As a final protection against unknown failures, we try to log all corner-cases, exceptions, and unexpected error events (using Rollbar). This lets us know immediately when an unexpected error is hit and the corresponding metadata allows us to reproduce the test using Assertible.


3. Insufficient manual testing

Manual testing is generally done incrementally throughout the lifecycle of an app. On many teams, developers manually test each new feature as it's deployed. Larger enterprises may have a QA team with a more specialized set of tools but the process is still the same. As new features are built, only the newest and most critical features are tested, leaving holes in test coverage.

QA Testing Manually

Problems with manual testing

  • Time consuming
  • Error prone
  • Difficult to measure

The process of manually testing a web app each time it's deployed works well for single features which are new, but doesn't scale well for non-trivial apps that change frequently.

For each feature and bug our team develops, we write a new test using Assertible. We execute these tests every single time we deploy our web app and API to a staging or production environment. These tests then continue running on a schedule.

This has significantly reduced the burden of testing an application with a larger surface area and growing number of features and dependencies.


Automated testing best practices

Here's a more specific breakdown of how we use Assertible to test and identify various deployment failure scenarios:

  • Run tests after each and every deployment

    Specifically, we use the Assertible GitHub deployment integration As mentioned above, we deploy all branches to a staging environment (as pull requests made through GitHub). Doing this maintains a robust continuous deployment pipeline where all pushes have several layers of testing; from unit tests to end-to-end tests.

  • Run tests on a schedule

    Along with running tests on every deployment, we also run tests frequently on schedules. In the future we plan to expand the capabilities to run different types of tests at different frequencies. For example, tests that create and delete data should be run less often on production systems.

  • Write tests for new features

    We create a new Assertible test for all new changes to our web app or API. A variety of workflows are tested and we're working towards having 100% API coverage.

    We do NOT consider end-to-end tests a replacement for testing before deployment. We believe testing immediately after building and before deploying is crucial to keeping software high-quality and should be the first line of defense against bugs and regressions, followed by other kinds of integration tests and Assertible tests.

  • Write a test for each new regression or bug

    We create new Assertible tests when we find bugs and regressions in our own code base. This helps us make sure they don't pop up again. We keep a record of all links to our issue tracker and pull requests in a test's description to conveniently lookup extended information about a specific test.

:: Christopher Reichert

Full Article


4 common API errors and how to test them

02/10/2017FeaturedCody Reichert

API failures are inevitable; the important part is how quickly you can react. Here are 4 common API failures, and how you can test them.

Preventing and reacting to problems in your web services is something every developer, QA analyst, and project manager should be thinking about often. As web services grow they require more complex types of testing, but there are a few fundamentals that should always be covered in your automated API tests.

  1. Invalid SSL certificates
  2. Service provider outages
  3. Too many redirects
  4. Invalid payload formats

Here's how you can continuously monitor your web service for these issues using Assertible.

Try Assertible for free to start testing your API

Invalid SSL certificates

It happens to the best of us: once a year an SSL cert expires and error reports start rolling in from API clients. These types of outages can be tough to track down and correct, so it's important to monitor your APIs SSL certificates and be alerted of any failures.

SSL validation testing with Assertible

Assertible has a core assertion to validate SSL every time a test is run. If the validation of your certificate doesn't pass, the AssertValidSsl core assertion will fail the test. This is configurable per environment, so if your production web service requires SSL, but your staging app doesn't, you can configure your tests to only validate SSL in production.




Service provider outages

Sometimes it's not our fault. Service providers go out too. A web service becoming unreachable can cause widespread issues. Monitoring your API with a ping can help spot these situations fast.

Frequent health checks with Assertible

A basic request to test for 200 status codes is a good way to ensure your service is reachable. You can add these tests to a schedule on your web service that runs on a given interval and alerts on any failures.


GitHub Web Service - Passing Assertible Tests

Too many redirects

Redirects in APIs and web applications are a part of how some services work, but if something spins out of control you don't want Chrome showing your users a TOO_MANY_REDIRECTS page instead of your app.

Enforce a redirect limit

The Max redirects setting on each test sets a limit of the number of redirects the test's request can make. If it exceeds that number, the test will fail. Any API or web app that utilizes redirects, should have these cases tested.


Configure max redirects for an API test

Invalid payload formats

Sometimes applications return data in unexpected formats, like when a JSON API responds with an HTML error page, and things start to go wrong. It's important to validate your APIs response data, in normal and error scenarios, so your clients don't start to fail.

Payload validation with Assertible

Assertible has a JSON validation and an HTML validation assertion to test the response of your API. Every time your test is run, the response body will be checked and you will (hopefully!) see a passing assertion:


JSON payload validation with Assertible


Having your bases covered with these steps is good start to correctly testing your APIs and web services. You can join Assertible for free and get started today!

Feedback or Questions? Tell us your thoughts on Twitter.

Assuredly yours,
Cody Reichert
Co-founder Assertible

Full Article


Testing an API behind a firewall

02/08/2017FeaturedChristopher Reichert

TL;DR Test APIs behind a firewall or on development machines using ngrok

Try it for yourself with a free Assertible account

1. download ngrok

The first step is to download a program to expose localhost (or another internal url) to the Internet. We'll use ngrok which is a program can be downloaded for free and available for all major operating systems.

https://ngrok.com/download

Download and unpack ngrok.

2. start ngrok

The next step is to expose your web service. If your web service is on localhost, simply run the following on the command-line:

$ ngrok http 3000

If you need to point ngrok to another host behind a firewall (e.g. inside an intranet), use the full host instead of a port only:

$ ngrok http internal.host:8099

You should see a line that looks something like this:

Forwarding    http://7e9ea9dc.ngrok.io -> 127.0.0.1:3000

or

Forwarding    http://7e9ea9dc.ngrok.io -> internal.host:8099

Copy the *.ngrok.io URL to your clipboard or write it down.

3. create a web service in Assertible

Next, create a web service in the Assertible dashboard.

Create an API pointed at an ngrok url

Use the ngrok URL obtained in step 2 as the host of your Assertible web service (e.g. http://7e9ea9dc.ngrok.io).

4. start testing!

Assertible will automatically run a test once when you create a new API to help you get oriented with the results.

ngrok Assertible test result

You should see a similar output in ngrok (although you may not have any failures!).

ngrok output

From this point, you are completely setup and free to configure Assertible to test various aspects of your API or web app. Have fun!

Resources and documentation


:: Christopher Reichert

Full Article


Creating a setup step: Using variables in your tests

12/28/2016EngineeringCody Reichert

Using setup steps to capture variables and parameterize HTTP requests allows you to test a wide range of use-cases for your web service. Here's how it works with Assertible.

Signup for a free account to start testing your API

When testing an API, there are often cases where you need to use data from another HTTP request in your test. For example, to test a /users/{{userid}} endpoint, you first need to call /users to get a {{userid}}.

Assertible's setup steps give you the ability to do this by capturing variables and substituting them in your test before it is run. Let's walk through creating a test for our /users/{{userid}} endpoint and using a setup step to populate the {{userid}}.

Outline of our process

  1. Create a new test for the /users/{{userid}} endpoint.

  2. Add a setup step that populates the {{userid}} variable.

  3. Use our variable to make assertions on the HTTP response.

If you haven't set up your web service yet, you can get started with that here.

When we're done, we'll have a test for a dynamic API endpoint with assertions using dynamic data:


Testing a JSON API with route parameters

Create a new test for your web service

First let's create new API test. On your web service's overview page, find the "Manage" dropdown and choose "Create test" (docs). In the new test's request configuration, set the endpoint as /users/{{userid}}:


Create an API test with Assertible

The double-brace syntax around the {{userid}} in the endpoint is a test variable. Before the test is run, the variable will be populated with the data from our setup step.

If we run our test now, it will fail with a 404 because the {{userid}} variable wasn't populated with any data. Let's move to the next step and get our test using the correct user id.


Make a setup step and populate a variable

Now that we have a test for our API using a variable, we need to populate our test variable with some data. There are a couple ways to do this with Assertible:

  • HTTP request setup step (docs)
  • Random variable setup step (docs)
  • Hard-coded test variables (docs)

For our /users/{{userid}} endpoint we'll use the first option. Our setup step will make an HTTP request, before our test is run, to call the API's /users endpoint and save an id from the response to the {{userid}} variable.

Create a setup step by visiting the "Setups" tab of your new test and choosing "Add setup step". Next, configure the setup step to make an API request and capture a variable from the response. Your settings should look like this:

  • Name: Fetch userid variable
  • Generator: Select the HTTP request generator
  • Request settings: In this area, fill out the URL and other settings to call the /users endpoint.
  • Variable capture: Add one variable capture where:
    • Source: Select Body
    • Name: Our variable name is userid
    • Selector: In the /users JSON response, we want to select the first users ID: .[0].login


Assert API test setup step - variable capture

Learn more about creating setup steps here.

When you have everything filled out, save the setup step and enable it for this test. Now the {{userid}} variable in our API endpoint will be replaced with data from the setup step! If we run the test now, our 200 OK status code assertion will succeed and the test will be passing.


Add JSON response assertions using variables

Prop tip! Variables can be used in assertions as well as test settings like an endpoint. Learn more.


With a passing web service test, we can start creating better assertions for the API response and make our test more robust. Using the JSON path data assertion, we will test that the /users/{{userid}} endpoint is returning the correct data for the userid parameter.

On the "Assertions" tab of your test page click "Add assertion", and for the type of assertion choose JSON path data. This assertion allows us to use the JSON path selector syntax to assert that the .id field in the JSON API response is equal to our {{userid}} variable used in the endpoint:


JSON API response assertion

The Assertion Comparator will stay as Equals and the the Assertion Target will be the {{userid}} variable. If you save the assertion and run the test now, you'll see that our assertion target is populated with the {{userid}} value.

Every time our API test is run, we are asserting that the /users/{{userid}} route returns the correct user data for the user id that was provided. The end result will look like this:


Assertible API test result with variable capture

Wrapping up

To sum up what's happening in our test:

  • Before the test is run, a separate HTTP request is made to populate the {{userid}} variable.
  • Any usage of {{userid}} in our test will be replaced with the captured data.
  • The test is run with the replaced variables in our endpoint.
  • Assertions are made on the JSON API response.

There's a lot more you can with setup steps, test variables, and JSON path data assertions. Check out more examples in the docs and create your free Assertible account to get started today!

If you have any questions or feedback, send us a message or reach out on Twitter!


Assuredly yours,

Cody Reichert
Co-founder, Assertible

Full Article


Why we test every single API deployment

10/25/2016EngineeringCody Reichert

At Assertible, we automatically test every API deployment. Here's why.

Having a reliable process in place for testing and validating API deployments provides a huge amount of confidence when deploying new changes into production. It's the best way to know that your application is ready for your users, and it saves developers time by decreasing the need for manual testing.

In this post I'll discuss our continuous testing process, and how we use Assertible to validate releases with post-deployment testing.

Assertible GitHub Status Check


How we continuously test

At Assertible, our continuous integration/delivery/testing setup is pretty standard:

Running API tests automatically after every deployment quickly validates and provides feedback about the release.

For example, if a pull request is opened, the code will be built and deployed to staging. API tests are automatically run against the new app on the staging environment.

If this new deployment doesn't pass the Assertible tests, the GitHub status check on the PR will fail and tell us the code shouldn't be merged.

Assertible GitHub status check failing


More on the Assertible GitHub integration

By using Github we're able to take advantage of the Assertible/GitHub integration and deployment events.

After the code is built and in the process of being deployed to staging, Assertible receives deployment status events. When the deployment is successful, Assertible runs all tests for that API and updates the status check.

The assertible/deployments repo has instructions on setting up GitHub deployments as a part of your CI with a simple script.

Another neat part of using GitHub status checks with Assertible is that we can see the result of our API tests on the master branch. Any commit made to the master branch means a deployment to production. By looking in the commit history we can see that the API tests were all successful.

Assertible GitHub commit status check


API testing with Assertible

All API tests cases live in Assertible. There are tests set up to validate all API endpoints, test core functionality, and make assertions on response data integrity.

These are some of the tests that are run when a new version of the API is deployed.

Assertible API Testing


Before continuous testing, these all had to be run manually. The manual process is obviously time-consuming and error prone -- no one wants to do that.

With a solid continuous testing workflow in place, manually testing these endpoints is unnecessary. I can sleep sound knowing that everything has already been validated and changes can be pushed to production with confidence.

If a test fails, the status check on GitHub will have a link directly back to the failing test result to easily see why:

Assertible failing test


Uh oh! It looks like we broke some authentication logic in the /tests handler. This is where continuous testing saves our skin and our time (and in some cases our job!).

This is why we test every. single. deployment.

If all of the API tests are passing, we can happily merge knowing that the release has been validated and tested.

Side note! Assertible is free. If you're interested in setting this up yourself, you can get started here.


Continue the discussion

I always enjoy hearing how other teams test and validate API deployments. Tweet at me (@CodyReichert) or us (@AssertibleApp) and let's see what you're working with!

Assuredly yours,

Cody Reichert

Full Article


GitHub integration status checks

07/29/2016FeaturedChristopher Reichert

Instantly view GitHub pull request status checks for your post-deployment tests

Try the GitHub integration with a free account

We are very happy to release GitHub status checks for our post-deployment testing integration. Using Assertible alongside GitHub deployments, you can test and verify that your staging or production web services are operating correctly with end-to-end tests.

GitHub Status Checks with Assertible

To get started, sign in with GitHub and check out the open source scripts in our deployments repo.

Resources and documentation

:: Christopher Reichert

Full Article


NPM Package: Lambda CloudWatch to Slack

07/26/2016EngineeringCody Reichert

This package provides a function to be run on AWS Lambda that sends CloudWatch notifications to your Slack channel. Check out the repo

Better Slack notifications for AWS CloudWatch

We released a new package for sending better AWS CloudWatch notifications to your Slack channel: lambda-cloudwatch-slack. Check it out on GitHub or NPM.

The repo provides a function that can be deployed to AWS Lambda and sends your CloudWatch alarms to a Slack channel. The README contains all the info you need on setting it up.

Improving default AWS messages

The default message aren't pretty and are tough to decipher. Using this function to send your messages improves on the default in several areas. Here's some screenshots:

AWS CloudWatch Notification for Slack


AWS CloudWatch Notification for Slack

Elastic Beanstalk notifications


Elastic Beanstalk Slack Notifications

Further improvements and contributing

If you have any feedback or ideas, feel free to open an issue on GitHub or send us a message on Twitter. Contributions are appreciated, you can open a pull request.


Assuredly yours,

Cody Reichert

Full Article


5 tools to complete your CI pipeline

07/20/2016FeaturedCody Reichert

TL;DR: Continuous development workflows make developers more productive by providing constant feedback during the development cycle. These 5 services will improve your CI pipeline.

Continuous development

Continuous integration, delivery, and testing provide teams with a way to frequently build new code, deploy changes to applications, and run tests before and after each step.

There are a few stages to a good continuous development pipeline:

  1. Integration: Build code and run unit and integration tests
  2. Delivery: Deploy ('deliver') your new application
  3. Testing: Automated testing of staging and production environments

This post will go over what happens in every step of a good continuous workflow. Most of the tools and processes mentioned below are geared towards hosted services, particularly ones that integrate with a GitHub deployment pipeline.

Continuous integration

Continuous integration is a process for frequently building and testing new code changes. In most cases, a central repository (eg, git) will have a master branch where all changes are merged and picked up by the CI build system.

With most tools, you will have a configuration file (eg, .travis.yml) describing how to build the application and run unit tests. As you push changes to a repository, your CI will build the code from your config.

A good CI pipeline will ensure that all code can be built and run and that all unit tests are passing before being merged into the main branch for deployment.

Tools

There are many great hosted CI services that work with your existing GitHub repositories. It may take trying a few to see what works best for you. Two of the most widely used services are:

Circle CI

Free and paid plans available. Website

The modern continuous integration and delivery platform that software teams love to use. - CircleCI

Travis CI TravisCI Logo

Free and paid plans available. Website

Easily sync your GitHub projects with Travis CI and you’ll be testing your code in minutes! - Travis-CI

Continuous delivery

Continuous delivery is the process of deploying new versions of an application frequently, sometimes several times daily, to staging and production environments. An important part of continuous delivery is that it makes application changes more reliable by by providing a consistent, reproducable pipeline for pushing new changes.

Deploying an application into a staging or production environment can get hairy, but a good delivery system will have the automation to make it simple. Being confident about rapidly pushing code and having a good deployment process can greatly improve a product.

Tools

Like CI, there are a lot different tools for continuous delivery. One of the most powerful is Heroku Flow:

Heroku Flow Heroku logo

Free and paid plans available. Website

Heroku Flow brings together Heroku Pipelines, Review Apps and GitHub Integration into an easy to use structured workflow for continuous delivery, so you can test early and deploy often, with better results for your users. - Heroku

If you're team is already using GitHub to host code, the Heroku GitHub integration provides tools for setting up staging environments on pull requests ('Review Apps') and then deploying those to production.

Continuous testing

The next step in a solid continuous development pipeline is testing. Usually, new code will be deployed to a staging location where changes can be tested and approved before reaching production. At this point, automated tests are run against the web service to ensure all services are working correctly.

A lot of times these same tests will be run against the production environment as well. A dedicated staging or review area is not required, but it does provide huge benefits. Another common work-flow is that all changes merged into master will be immediately deployed to production. In these cases, your continuous testing should run as soon as a new application is deployed.

Tools

Tools for continuous testing can range from manual QA teams and custom scripts for monitoring or testing API endpoints. Assertible is one hosted service that can run post-deployment from your CI pipeline.

Assertible

Free and paid plans available. Sign up for a free account

Teams and individuals use Assertible to test and monitor production web services, create scheduled assertions, and gain confidence in crucial infrastructure. - Assertible

You are on the Assertible Blog so I might be a bit biased. Post deployment tests are, however, a crucial part of a good continuous development workflow that will enable you and your team to write more, better code.

Wrapping up

What tools does your continuous development pipeline consist of? Shoot us a message on Twitter and let us know!

The assertible/deployments contains more information on integrating continuous testing into your existing workflow. Check it out and sign up for a free Assertible account to start testing your deployments!


Assuredly yours,
Cody Reichert

Full Article


Creating email templates with React components

07/18/2016EngineeringCody Reichert

TL;DR - This post discusses and provides code samples for how we use React components to create our email templates.

The beauty of React.js is in the reusability and composition of components. Components are encapsualted pieces of UI that can be composed to build larger interfaces. At Assertible we use React.js for pretty much everything that you see: website pages, the entire dashboard, our email templates, and even this blog!

Emails are tough, so this post hopes to give a few pointers on how you can apply style attributes (for example, <p style={{...}}>) as well as classes (eg, <p className="..") to your React components, which allows for consitent rendering across email clients. Note that a lot of the information below is specific to webpack and babel.

The problem with emails

If you've ever written email templates, you know the pain of trying to get a consistent output in different email clients. Some will recognize <style> tags, while others will strip them; some require inline styles while others we still have no idea what they want.

This post provides an approach for using React.js components to create a module that be rendered to static markup (renderToStaticMarkup) and used in emails while keeping a consistent render output across clients. I won't cover all the caveats of email client's here, but if you're interested MailChimp has a good overview on the topic.

The goal output

In order to have a good output across email clients, we want to have our final markup contain both style attributes and class attributes. The class attributes are optional, since all the styles are inline as well, but I've found that it prevents some corner cases and it doesn't add too much bloat.

Implementation

Now that we know we need both style attributes and classNames, we can come up with a goal before and after output for our markup.

This can be improved, of course, but ideally we can use the JSX spread operator to apply everything we need to an element at once. Something like this will work:

Before

<p {...style(['.text-muted', '.pull-xs-right']}>
  ...
</p>

The style function will take a list of 'classes' (classes that should be found within a CSS file -- more on that below), and apply both the styles and classes to our element:

After

<p class="text-muted pull-xs-right" style="color: gray; float: right">
 ...
</p>

With this approach we can guarantee a consistent render output across email clients. Gmail will be happy with the inline styles, and all the other clients will use whichever they prefer, classes or inline style attributes.

Let's go over the style function and what's happening:

style( Array<string> ): Object

The style function provides most of the functionality for applying the styles. Here is the entire code for style module (comments are important):

const bootstrap = require('/path/to/bootstrap.css')

export const applyStyle = (selectors, style = {}) => {
    // Our 'className' will be appended here
    let className = ""

    // For each 'selector' provided (eg '.pull-xs-right'), look up
    // the styles in the stylesheet. 'inline-style-loader', described
    // more below, is how we query the stylesheet.
    for(const key in selectors) {
        const slctr = selectors[key]
        const rules = bootstrap[slctr]
        // If CSS rules exist for this selector, add them to
        // 'className' and 'style'.
        if(rules) {
            const styles = rules.split(";")
            for (const key_ in styles) {
                const [attribute, value] = styles[key_].split(":")
                const attr = attribute.replace(/-([a-z])/g, g =>
                    g[1].toUpperCase()
                )
                // add key/value to the 'style' object
                style[attr.trim()] = value.trim()
            }
        }

        // Append selector to className
        className += ` ${slctr.substr(1)}`
    }

    // Return an object that can be spread directly onto a component.
    return {
        style,
        className
    }
}

If the comments don't provide enough info, the basic gist is: style takes an array of class names and returns an object with style and classNames ({ style, className }) that can be applied directly to a React component. The class names arguments are looked up in a style sheet (bootstrap in the example above) that is imported with inline-style-loader (discussed more below).

We now have a function that does this:

const styles = style(['.pull-xs-right'])
// {
//   className: "float:right;",
//   style: { float: 'right' },
// }

inline-style-loader

If you're wondering how we're able to query the CSS file as a JSON object to get access to styles, that is due to inline-style-loader. This webpack loader allows you to import a CSS file and access it as a JSON object.

One problem is that inline-style-loader will also format the styles to be used in a style _attribute (style="..."), so in the style function we have to to some string splitting.

inline-style-loader can be added to your webpack config like this:

module.exports = {
    // ... the rest of your config
    module: {
        loaders: [
            // ... the rest of your loaders
            {
                test: /bootstrap.css$/,
                loader: "raw!inline-style"
            },
        ]
    }
}

Wrapping up

Using this method has been successful for us so far. We use this for all of our emails sent from the dashboard as of now. Here's an example signup email rendered in Gmail:

Assertible Getting Started Email



Try it out

Teams and individuals use Assertible to test and monitor production web services, create scheduled assertions, and gain confidence in crucial infrastructure. Sign up for a free acount today.

Sign in with GitHub


Assuredly yours,
Cody Reichert

Full Article


Continuously testing Heroku GitHub deployments

07/11/2016FeaturedChristopher Reichert

TL;DR: Run tests against your web service every time you deploy a new version of your application from Heroku. Try it free.

Heroku logo

Setup

  1. Configure GitHub to connect w/ Heroku

    In the Heroku dashboard, navigate to the app you would like to test and click the Deploy tab.

    Heroku dashboard

    In the Deployment method section, select GitHub

    In the Connect to GitHub section, select a repository (you may need to click Search, see the image below).

    Heroku choose GitHub repository

    Once heroku recognizes your repo, click Connect. This should enable new options titled Automatic Deploys & Manual Deploys. The heroku dashboard should indicate that GitHub is now connected. For more information about setting up Heroku w/ GitHub, click here.

  2. Connect Assertible w/ GitHub

    If you don't already have an Assertible account, you can get started by clicking the link below to sign in with GitHub.

    Sign in with GitHub

    Once you create an account, create your first Assertible test by entering the URL of a web service or importing a Swagger spec. Once you have created a test, navigate to the Deployments tab. Click the Add to GitHub button found on the GitHub integration and select a repository.

    Fig 1.0 GitHub Deployments Integration

  3. Deploy your web service

    At this point, Assertible is completely configured to receive events from GitHub when your app is deployed. To test the integration, you can push code or do a manual deployment from the Heroku dashboard.

Resources and documentation

:: Christopher Reichert

Full Article


Effortless post deployment testing with GitHub

07/10/2016FeaturedCody Reichert

Run tests against your web services every time you deploy a new version by connecting Assertible to your GitHub account. Try it free.

Assertible now connects with your GitHub account to run your test suites after deploying a new version of your application. Testing new features and deployments are a huge and crucial part of any developers workflow who runs web services. We aim to make this process better and easier, so we have a 3 step process for automatically testing new deployments:

How it works

Test deployments with Assertible


Connect Assertible with GitHub

Assertible connects to your GitHub repository to receive deployment events. You can connect Assertible with your GitHub account by clicking on the button below:

Sign in with GitHub

Integrate with CI

Assertible tests will start running after a succesful deployment. A lot of continuous integration and deployment providers will work out of the box without any other configuration. For example, see how heroku deployments work with Assertible.

Example CI configurations

The assertible/deployments repository contains example configurations for popular CI services. Check out the repo to see how Assertible works with your CI provider.

Deploy code, run tests

Once you've got your GitHub account connected, and your deployment pipeline is set up to send events then you're all set! You can now start creating tests for your web service, making assertions, and setting up Slack notifications.

Resources and documentation

If you're ready to get started setting up your post deployment tests, check out these resources:

Assuredly yours,

Cody Reichert

Full Article


Simplified Assertions for Your Web Services

06/26/2016EngineeringCody Reichert

Testing your API shouldn't be harder than writing it.

Today we're happy to release new assertions for testing your website and API. Testing can quickly become complicated, but it shouldn't be. Assertible's simplified assertions allow you run known-to-be-valuable tests against your live web service. We've split the new assertions up into a couple of groups: Website 101 and API 101. To try these out on your own website, sign up for a free account and enter a URL.

Assertible - Simplified website testing

New Assertions

All of these links go to the documentation

Website 101

API 101

Simplifying assertions and how they're created makes your tests, and your time, more valuable. All of the new assertions can be enabled with just a click and some can take additional configuration options to make more specific assertions.

After running your test, you'll see the results of each assertion on the Test Results tab:

Assertible - Web service testing results

Create a free account and try it out!

Assuredly yours,

Cody Reichert

Full Article


Introducing Assertible

04/25/2016FeaturedCody Reichert

We're proud to announce Assertible, a tool for testing and monitoring production web services.

What is Assertible?

Assertible is a testing and monitoring service for websites and API's, primarily focusing on three goals:

  • Easily create robust and meaningful assertions for your web app.
  • Monitor your web services and notify you and your team when things aren't working as expected.
  • Integrate with familiar third-party services (like Slack and GitHub!) so you can immediately reap the benefits, however you like to work.

Getting started

The easiest way to get started is to import a Swagger spec (Swagger spec). The video below shows a quick demo setting up tests for the Hacker News API using a Swagger spec.


Learn more

Sign up for a free account and try it out! You can start adding your own websites, get started with some of the demo API's, or add tests for any other service.


We would love to hear any of your feedback, you can reach us on the contact page.

Assuredly yours,

Cody Reichert

Full Article