Testing and validating API responses with JSON Schema : Assertible

10/12/2017 Featured Cody Reichert

Testing and validating JSON APIs is an important aspect of running a quality web service, but managing checks for large and diverse response bodies can be difficult. Using JSON Schema to construct a model of your API response makes it easier to validate your API is returning the data is should.

API consumption is on the rise everywhere from businesses to personal projects, and they're constantly being changed to support new features, fix bugs, and change functionality. If you're tasked with testing an API, or you've hit breaking-change issues in a service you work on, you already realize the importance of having a definition, a schema, that describes the data your service will accept and the data it will return.

JSON Schema is a great tool for the job. In this post I want to illustrate how you can continuously test and monitor your API responses, ensuring they adhere to a specified format, and get alerted when breaking changes occur.

What is JSON Schema
Basic JSON Schema example
JSON Schema assertions in Assertible

What is JSON Schema

First, let's take a look at what JSON Schema is and what it can do. This is the definition from json-schema.org:

JSON Schema is a vocabulary that allows you to annotate and validate JSON documents. It describes your existing data format with clear, human- and machine-readable documentation for complete structural validation, useful for automated testing and validating client-submitted data.

json-schema.org

In simpler terms, JSON Schema is a contract for your JSON document that defines the expected data types and format of each field in the response. For example, you can create a schema that says your API's /users endpoint returns a top-level array of objects (eg, users). Naturally, you can recurse into each user and describe the format of its fields as well.

There are many tools and libraries available to create and work with JSON Schemas, making getting started easy. In this post, I'll walk-through an example of using Assertible to create a test that validates an API response against a JSON Schema. From there, automated monitoring and alerts can be set up for continuous testing.

It's worth investigating other tools available, especially libraries in your programming language of choice. JSON Schema is a robust tool that has many application useful for all sorts of validation and documentation.

Basic JSON Schema example

It's helpful to see an example of how a JSON Schema is structured. Expanding on the /users API example from the previous section, imagine the JSON response is structured like this:

[
    {
        "firstName": "John",
        "lastName": "Doe",
        "age": 35
    }
]

The structure of this JSON response has a few important things to point out, let's dissect it:

The API returns a top level array
Each array entry is an object (a user).
Each user has a firstName, lastName, and age key.
The firstName and lastName are strings, while the age is a number.

Each of the points mentioned above can be described in a JSON Schema. That schema can then be used to validate that an actual JSON response from the API is compliant with the definition.

The following JSON Schema definition describes the response above:

{
    "$schema": "https://json-schema.org/draft-04/schema#",
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "firstName": {
                "type": "string"
            },
            "lastName": {
                "type": "string"
            },
            "age": {
                "type": "integer"
            }
        }
    }
}

Reading through the JSON Schema line by line, you can clearly see how it represents the shape of the /users response. It details the type and structure of the entire body, as well as each individual field. To go over a few points:

The top-level type key tells us the API returns an "array".
The items describes the shape of each array entry (each user).
The firstName and lastName fields are strings, while age is an integer.

This is a simple example of a JSON Schema, but the usefulness is clear. It's a great tool for documenting your API responses and has a lot of benefits and use-cases that can be adapted.

For more examples, check out the JSON Schema examples page and getting started guide.

More complex JSON Schema

In addition to defining the shape and data type of the JSON, you can also use things like format to state a specific field should match a regex. Additionally, there is support for things like examples and descriptions, which can be a great way to create a definition and documentation at the same time. The documentation on GitHub has a great overview of the specification.

In this post, we're primarily concerned with making sure the actual API response matches this schema. The next section will describe how to use Assertible to make this check.

JSON Schema assertions in Assertible

There are many tools and libraries available to test API responses against a JSON Schema. Assertible has an easy to use JSON Schema assertion to validate an API response that we'll set up in this example.

If you haven't gotten started with Assertible yet, you can create a free account and enter your API's URL to get started. For more help on this, check out the quick start guide.

Verifying an API response against a JSON Schema in Assertible is only a couple of steps. Let's walk through it:

First, create a test that calls the API endpoint you want to check
On the test's page, navigate to the Assertions tab, select Add new assertion, and choose the JSON Validation assertion. You will see a form like this:

On the left side is the form to enter the JSON Schema document, and on the right side is the API response. If the API response does not match the JSON Schema when the test is run, the test will fail.

And there you have it! A simple way to check your JSON API responses against a JSON Schema. The beauty of the JSON Schema assertion is that the schema can be as simple or complex, as thorough or as sparse, as you need. You may only want to check one or two fields, or you can check the entire document.

Automation and alerts

With Assertible, it's easy to put this test on a schedule for automated testing and set up failure alerts.

On the Assertible free plan, you can set up an hourly test schedule, and on one of the upgraded plans, you can set up 1 minute, 5 minute, and 15 minute schedules to make sure your API is continuously tested.

All alert and notification functionality is available on Assertible's free plan. You can send failure messages to your email with hooks or to a team chat using the Slack integration.

Other tools

Assertible is one way to run this test on your API, but of course there are many other tools for working with JSON Schema, especially as programming language libraries.

JSON Schema Test Suite is a great open-source project that contains a large list of JSON Schema parsers for almost every language, which can serve as a good starting point. On NPM there are several widely-used packages for working with and testing JSON Schemas.

In any case, this type of testing is valuable and I highly recommend integrating it with your QA and testing process. It helps cover large amounts of data quickly and removes the need to create custom checks for each API endpoint, response, and field.

Did I miss a JSON Schema project? Let me know and I'll add it!

Conclusion

Testing your API responses against a JSON Schema is a very efficient and effective way of ensuring your APIs are working as expected. The benefits of this are wide-ranging: less bugs, better documentation, valuable tests, and happy users.

Additional resources

There are a few other great blog posts I can across while researching this topic that are worth reading:

If you have any questions, comments, or feedback on the post, I'd love to hear from you! Find us on Twitter or reach out any time and let's chat!

:: Cody Reichert

Testing and validating API responses with JSON Schema