How it’s built: SingleHop API Testing

Here at SingleHop, we have 3-4 major systems that interact with each other constantly. These systems are:

  • Manage (Our primary administrative interface and automation codebase)
  • DropZone (Our customer-facing API)
  • LEAP3 (Our client portal, you should know this one!)

Most days, the system looks like this:

How it All Works Diagram

You’ll notice that all of these services either expose or consume REST APIs in order to get their work done. In order to make sure that these systems continue to run happily while we’re regularly updating them, we need to test our changes before we push them out into production. One of my jobs the last several months has been to improve and expand the monitoring of these systems so we can catch any failures early and often. While there are many components to monitoring a distributed system like SingleHop's, I’m going to walk you through one tool that I've built in order to monitor our systems.

The APITester.

The APITester is a set of Python scripts (one of our in-house languages) that essentially just loops through our API calls, hits them while pretending to be a client, and logs the results. The final tally of Successful calls / Total calls is recorded, and if it’s less than 100%, throws off a notification that something went wrong.

How does the APITester know what calls to make? We tell it. Here’s a snippet of the configuration file for the Server controller in DropZone:

config:
  controller_root: server/
  logging: console
  auth: basic
  username: <client_id>
  password: 

calls:
- list
- list/detailed
- view/19086
- network/19086
- hardware/19086

And here’s a snippet of the debug output:

apitesters  : INFO     Attempting url https://dropzone.singlehop.com/server/list/
ok
apitesters  : INFO     Attempting url https://dropzone.singlehop.com/server/list/detailed/
ok
. . .
apitesters  : INFO     Attempting url https://dropzone.singlehop.com/server/hardware/19086/
ok
{'failure': 0, 'success': 5}
apitesters  : INFO     5 of 5 functions run in 4.63457393646 seconods
apitesters  : INFO     Complete!

This is a success! Note, this is running on our production systems, so success is the expected result. If this were running on our development environment, we would consider this code ready to be pushed out to production.

What happens if the script detects a failure? For reference, here’s another snippet, showing two failures.

. . . 
2013-09-03 11:09 requests.packages.urllib3.connectionpool INFO     Starting new HTTPS connection (1): dropzone.singlehop.com
2013-09-03 11:09 apitesters   ERROR    API call on URL https://dropzone.singlehop.com/server/network/19086/ not successful
2013-09-03 11:09 apitesters   INFO     Attempting url https://dropzone.singlehop.com/server/hardware/19086/
2013-09-03 11:09 requests.packages.urllib3.connectionpool INFO     Starting new HTTPS connection (1): dropzone.singlehop.com
2013-09-03 11:09 apitesters   INFO     3 of 5 functions run in 4.63457393646 seconods
2013-09-03 11:09 apitesters   INFO     Complete!

You’ll notice the report at the end of the script has changed to show that not one, but two calls failed. This would immediately signal to our QA team that this code needs to be sent back to the development team for an investigation of why these API calls failed.

We’re still in the process of rolling out this tool across all of our applications and production infrastructure, but the results so far have been very promising.