Making assertions

Assertions allow you to validate HTTP responses in your API tests. For example, a basic assertion would ensure that a request to assertible.com returns a Status code 200 every time -- or that the homepage contains the text Assertible.

Assertions are checked every time a test is run. The result of each assertion, and the reason it passed or failed, can be found on the Test Result page. Assertions are also used to determine a test's state. If any of a test's assertions are failing, the test is in a failing state.

Sections

• Add a new assertion
• Types of assertions
  • Status code
  • JSON path data
  • JSON validation
  • XPath XML data (XML/HTML)
  • Response header
  • Response time
  • Response text
  • HTML validation
  • Link checker
• Built-in core assertions

Add a new assertion

Assertible offers a selection of assertions that are commonly used to test HTTP services. For example, Assertible has assertions to validate HTML and JSON or check a webpage for broken links.

Each test configuration view has an Assertions tab. From there you can add, update, and delete any assertions for that test:

Types of assertions

Most assertions can be enabled with a single click. Some assertions offer additional configuration and customization. The available assertions are described below:

• Status code
• JSON path data
• JSON validation
• XPath XML data (XML/HTML)
• Response header
• Response time
• Response text
• HTML validation
• Link checker

Assert status code

Assert the desired HTTP status code of the test's request. The default is 200. This assertion is enabled by default for all new tests and can be configured to any value.

Assert JSON data

Asserting JSON response data allows you to select parts of a JSON response and make assertions using JSON Path syntax; inspired by the original JSONPath implementation.

For example:

Sample JSON

{
  "id": 1,
  "org": {
     "name": "Acme",
     "founded": 2015
  },
  "store": {
    "book": [
      { "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      { "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95,
      "parts": [
        {
          "id": "tire",
          "psi": "100"
        },
        {
          "id": "handlebars",
          "style": "drop"
        }
      ]
    }
  }
}

Given the sample JSON above, the following table shows the Assertible JSON Path data assertion configuration and result:

Note that if your JSON Path expression attempts to select a node that does not exist, the null value is not returned. In practice, this means your assertion will fail (at the time of this writing, Assertible does not support an "assert value exists" comparator but we're working on it!). For example:

Assertible's JSON Path syntax does support wildcard path selections. However, only single node selections are valid at this time. See the JSON Path wildcard syntax docs for more detail:

Note that .store[] is a JSON object wildcard which returns an array with all the top-level JSON objects at $..

Read more about the JSON Path syntax in the JSON Path wildcard syntax docs.

Assert JSON validation

Validate that JSON response bodies are valid JSON. Optionally, a JSON schema can be specified.

If no JSON schema is supplied, the JSON validation assertion will check that the HTTP response value is valid JSON. A top-level object, array, or scalar value will pass the test. Any un-parseable JSON errors or invalid values will fail the assertion.

JSON Schema Validation API

Assertible hosts a free JSON Schema Validation API service. You can POST your JSON Schema spec and JSON documents to validate and debug validation errors.

The API is simple. Using this curl command:

$ curl -XPOST "https://assertible.com/json" -d'{
  "schema": {"type":"object"},
  "json": {}
}'

The API will response with:

{"errors":[],"valid":true}

See the JSON Schema Validation API webpage for more details.

JSON schema

The JSON validation assertion also supports an optional JSON Schema Draft 4. When a JSON schema is supplied, the JSON validation assertion will validate the HTTP response data against the schema and report all validation errors.

There are some limitations with this assertion. A few json-schema features are not yet supported:

• format

  For example, the JSON string "chris" with a validation schema that uses format will ignore the formatting and validate the string:

    {
        "type": "string",
        "format": "email"
    }

• ECMA-262 regex

  Some parts of ECMA-262 regular expressions are not supported:

  • [^]: a validator with {"type":"string", "pattern": "[^]"} will fail to compile due to being an invalid regex.

• zero terminated floats

  For example, a validator {"type":"integer"} will work on 1.0 even though there is a trailing slash.

Speak up! Let us know if any of these unsupported features are valuable to your use-case. We are working on improving support for JSON schema but feel free to contact us.

Create JSON schema assertions for OpenAPI v3 responses

If you use OpenAPI v3, Assertible can automate the creation of JSON Schema assertions for every operation that has a valid response with a corresponding schema. Assertible checks the most commonly used response formats first. For example, a 200 OK response with an application/json media type will be used first. If there is not a response defined with a successfuly status code, the default response is used, assuming it includes a schema.

OpenAPI v3 JSON Schema compatibility support

OpenAPI uses an extended subset of JSON Schema to define data types in an OpenAPI definition. Assertible has support for the following OpenAPI schema keywords:

• nullable

  When nullable: true, assertible automatically translates any corresponding schema fields to the JSON Schema Draft 4.

  • type: "string" is converted to type: ["null", "string"]

  • type: [object,array] is converted to type: [null, object, array]

  • If enum exists, null is appended to the list of enums.

    Here's some GitHub issues that exhibit some of the confusion around these fields:

  • https://github.com/OAI/OpenAPI-Specification/issues/1368

  • https://github.com/swagger-api/swagger-ui/issues/4119

Assert XML/HTML data

Asserting XML response data allows you to select parts of an XML or HTML response using XPath and make assertions on the selected value. For example, a key id has a value equal to 1. The XML data assertions works well with both HTML and XML data sources.

Sample XML XPath assertions

See more examples and information about the syntax below:

<?xml version="1.0" encoding="UTF-8"?>
<store>
  <id>1</id>
  <org>
    <name>Acme</name>
    <founded>2015</founded>
  </org>
  <bookstore>
    <book category="cooking">
      <title lang="en">Everyday Italian</title>
      <author>Giada De Laurentiis</author>
      <year>2005</year>
      <price>30.00</price>
    </book>
    <book category="children">
      <title lang="en">Harry Potter</title>
      <author>J K. Rowling</author>
      <year>2005</year>
      <price>29.99</price>
    </book>
    <book category="web">
      <title lang="en">XQuery Kick Start</title>
      <author>James McGovern</author>
      <author>Per Bothner</author>
      <author>Kurt Cagle</author>
      <author>James Linn</author>
      <author>Vaidyanathan Nagarajan</author>
      <year>2003</year>
      <price>49.99</price>
    </book>
    <book category="web">
      <title lang="en">Learning XML</title>
      <author>Erik T. Ray</author>
      <year>2003</year>
      <price>39.95</price>
    </book>
  </bookstore>
</store>

Given the sample XML above, the following table shows the Assertible assertion configuration and result:

Assert response header

Make an assertion against a response header. In the header name field, specify the name of the header to look for. The target value field is dependent on the type of comparator used. For example, when using Contains, the target value only needs to match some part of the header value.

Assert response time

Asserts that the response time of the entire request is less than a specified value, 1 second by default. If the HTTP request takes longer to respond than the target value, the test will fail. Otherwise it will pass.

Assert text body

This assertion checks for any text in the response body of the request. The default value to check for is DOCTYPE -- you can set the target to any given text and assert the response body contains that string. If no match is found the test will fail and if a match is found it will pass.

Note: the body can be any Content-Type, but will it will be checked as a raw string.

Assert HTML validation

Asserts that the response body contains valid HTML, XHTML, etc markup per the WC3 specification. This is useful for finding invalid DOM and HTML errors in your webpages that may prevent them from loading correctly on some clients.

Assert link check

When enabled, this assertion will check all of the links on an HTML page and ensure a request to that target is valid. If any of the links on the page are broken, return a Status Code 404, etc, the assertion will fail.

Currently support html tags:

• <a>
• <script>
• <link>
• <img>

Built-in assertions

In addition to user-defined assertions, Assertible has a set of built-in assertions known as core assertions. These assertions deal with the default behaviors a test run must satisfy. For example:

• Assertible ensures that an HTTP request connects and executes properly: HTTP Request Success.

• Assertible ensures that a valid web service environment is loaded.

• Assertible ensures that SSL connections are properly validated, depending on the execution environment's Validate SSL settings.