Saturday, November 14, 2015

Swagger for the EVE XML API? Not so much...

There are plenty of third party libraries for accessing the EVE XML API in a variety of languages, some of which I have (or will) reviewed on this blog.  That said, each library has its own idiosyncrasies and each implementation takes a slightly different view of how to handle errors, package results, etc.  Each library also differs in the level of documentation provided, with the default usually being none.

Wouldn't it be nice to have a way to describe the EVE XML API in some API description language, then have a tool which generates bindings in several different languages?  Well yes, and such things have existed in computer science for many years (anyone else remember CORBA IDL?).  A recently popular library for doing this is Swagger, which is designed to model REST APIs using an extension of the JSON schema specification.  Swagger has an impressive set of tools for online editing of a REST API, trying out an API out against a live site, and generating clients (and servers!) in a variety of languages.  Also, they seem to claim in various places that XML is supported.  This sounds exactly like the right thing to try for the EVE XML API.  Except it doesn't work.  The rest of this blog talks about my attempt to make this work and ultimately what failed (short version: none of the generated clients support de-serializing XML documents yet).

Resources

Swagger Site
Sample Swagger spec for the EVE XML API

A Quick Intro to Swagger

In a word, Swagger is an API tooling ecosystem designed for REST based APIs.  Swagger is open source with the core libraries and tools all available on GitHub.  A company called SmartBear (of Code Collaborator fame) is behind much of the Swagger code, but there are many open source contributors as well.

Swagger takes as input an API spec written in a variant of JSON schema.  The spec itself can be represented as either YAML or JSON (these are largely interchangeable).  There are three main tools which consume Swagger API specs:

  • Swagger Editor: This tool lets you create an API spec online, test it out, then export a client or server backend in a variety of languages.  We use this tool below to create our EVE XML API spec.
  • Swagger UI: This tool turns an API spec into online documentation.  There's an online demo which lets you pull up any publicly available Swagger spec.
  • Swagger Codegen: This tool converts an API spec into a client or server backend in a variety of languages.  The Swagger Editor already has this functionality built in, so normally you'd only use the code generator directly if you need more control over the process, or if you plan to write your own generator.  If you plan to use your API mainly from the browser or other Javascript environment, then you don't even need to export a custom client.  Instead, you can use swagger-js which reads any API spec and provides the appropriate calls.  We show how to use this below as well.

The online tooling and capabilities of Swagger look very impressive.  This would seem to be a great way to document and provide endpoints for the EVE XML API.  Let's see how that goes...

Let's Make a Model for the EVE XML API

To get started, let's try making a model for the EVE XML API.  The model will define all the operations we can invoke, the arguments for those operations, and the structure of the return values.  We'll use the Swagger Editor to create and test our spec.  You can download the editor and run it locally, or you can also just run the live demo which is usually easier.  Here's what the editor looks like:

Swagger Editor Live Demo
On the left is a text editor where you create your API spec, and on the right is a live rendering of your spec including tools to try out your API calls.  There are numerous examples you can view from the File...Open Examples menu.  The menu also contains drop downs for generating clients and server back-ends in various languages.

To start out, we'll implement the ServerStatus API call.  This is a very simple call which takes no arguments and requires no API key.  We'll start with a clean slate, then build up the API file.  So we'll select File...New in the Swagger Editor to start with a basic template.  If you want to skip ahead a bit, you can just download the swagger.yaml file directly.  Or keep reading to understand how we're building this file from scratch.

The first part of the configuration is preamble describing the API, giving its location, the schemes we can use to access the API, and the type of data the API will produce:
swagger: '2.0'
info:
  title: Eve Online XML Endpoint API
  description: Eve Online XML Endpoint API
  version: 1.0.0
host: api.eveonline.com
schemes:
  - https
produces:
  - application/xml
Most of these settings can be overridden for each API call.  This isn't necessary for the EVE XML API since every call has the same basic properties.  So in this case the preamble says that all calls will hit api.eveonline.com using the https scheme, and every call is expected to produce xml output.

Next, we need to describe the actual API endpoints.  These are called "paths".  In our example, we're calling ServerStatus, so the complete path specification will look something like this:
paths:
  /server/ServerStatus.xml.aspx:
    get:
      summary: Current Tranquility server status
      tags:
        - Server
      responses:
        200:
          description: Server status
        400:
          description: Error
The "paths" keyword marks the start of the paths section and is followed by entries which give the actual path, the HTTP methods which may be invoked on that path, and the responses to expect.  In this example, the path is "/server/ServerStatus.xml.aspx" which is appended to the host (in the preamble) to give the complete path of this call.  We call this path with the "get" HTTP method, and expect two basic responses: 200 (OK) or 400 (Bad Request).  These responses are specified in the "responses" section.  The "tags" keyword is used to group this call for client generation and documentation purposes (we'll show that further below).

At this point, we actually have a usable spec which you can use to call the API.  If you're following along in the editor you'll see something like the following in the right panel:
Call Test Panel
If you click on "Try this operation" and hit "Send Request" (and you're connected to a network) then you should see a successful API call (click on "Pretty" or "Raw" in the output panel to see the XML output).

This is nice, but not much use without something which parses the response and gives us a nice structured object.  Swagger lets you do this by defining a schema for each response.  For the ServerStatus call, there are two types of response we need to handle.  A success response will look like this:
<?xml version='1.0' encoding='UTF-8'?>
<eveapi version="2">
  <currentTime>2015-11-05 23:22:36</currentTime>
  <result>
    <serverOpen>True</serverOpen>
    <onlinePlayers>20930</onlinePlayers>
  </result>
  <cachedUntil>2015-11-05 23:24:19</cachedUntil>
</eveapi>
It's also possible to receive an error response, which will have this form:
<?xml version='1.0' encoding='UTF-8'?>
<eveapi version="2">
  <currentTime>2015-11-05 13:16:33</currentTime>
  <error code="106">
  Must provide userID or keyID parameter for authentication.
  </error>
  <cachedUntil>2015-11-06 01:16:33</cachedUntil>
</eveapi>
Note the common elements in both responses.  Ideally we'd like our schema to reflect these common elements as well.  Here's one way to specify such a schema in Swagger:
definitions:
  ServerResponse:
    type: object
    xml:
      name: eveapi
    properties:
      version:
        type: integer
        xml:
          attribute: true
      currentTime:
        type: string
      cachedUntil:
        type: string
  ErrorResponse:
    allOf:
      - $ref: '#/definitions/ServerResponse'
      - type: object
        properties:
          error:
            type: string
            properties:
              code:
                type: integer
                xml:
                  attribute: true
  ServerStatus:
    allOf:
      - $ref: '#/definitions/ServerResponse'
      - type: object
        properties:
          result:
            type: object
            properties:
              serverOpen:
                type: boolean
              onlinePlayers:
                type: integer
Swagger schemas live in the "definitions" section of the specification.  We first define a ServerResponse type which captures the common elements of all responses (version, currentTime and cachedUntil).  XML definitions in Swagger require a bit more work because XML allows both attributes and values.  In the case of the ServerResponse, we use the "xml" tag to indicate certain XML features.  The "name" tag defines the name of the root tag for the ServerResponse (i.e. eveapi).  The "attribute" tag indicates which properties are XML attributes.  One more detail: the Swagger date-time type can't parse EVE XML API time values, so we have to use "string" for those values instead.

We'll use the ServerResponse schema as a base definition for two additional schemas: ErrorResponse and ServerStatus.  The Swagger tag "allOf" is a schema composition operator.  This operator accepts a list of schema object definitions which are combined to form a single schema.  For example, the ServerStatus schema specification says to combine all of the fields of ServerResponse with the schema which defines a "result" property, and two additional properties for serverOpen and onlinePlayers.  Note the use of the $ref operator to pull in the definition of the ServerResponse schema.

The last step is to add the appropriate schemas to the response section for the ServerStatus call.  The new "paths" section now looks as follows:
paths:
  /server/ServerStatus.xml.aspx:
    get:
      summary: Current Tranquility server status
      tags:
        - Server
      responses:
        200:
          description: Server status
          schema:
            $ref: '#/definitions/ServerStatus'
        400:
          description: Error
          schema:
            $ref: '#/definitions/ErrorResponse'
Definitions done, let's see if we can make this into a real XML API client.

Nice Model...Let's Make a Client!

If you're using the Swagger Editor with our example, then at this point you have a usable spec which you can try out.  However, try as I might, I could never get the Swagger Editor to render the response objects (as displayed in the "Rendered" tab of the response when you try the API).  That was my first clue that maybe this wasn't going to work so well.

I decided to press on and generate a client.  Maybe it's just the Swagger Editor that is broken, right?  The easiest client to generate is a Javascript client.  This is easy to do because the swagger-js project does it for you on the fly.  There are a few different ways to test this out.  I decided to use the NodeJS approach as documented on the site.  The following code snippet does the trick:
var client = require('swagger-client');
var swagger = new client({
  url: 'http://localhost/swagger.json',
  success: function() {
    console.log('swagger ready');
    swagger.Server.get_server_ServerStatus_xml_aspx({}, {}, function(data) {
      console.log(data.obj);
    });
  }
});
There's one catch here in that the Javascript client wants to load your Swagger spec from a url.  To make that work I used my locally installed web server and put my spec there.  The nice trick with the swagger-js client is that the same code works for any valid Swagger spec.  So there isn't a separate generation step, you just drop this code into your app and you're good to go.

If you run this, however, you won't get what you expect because "data.obj" will be null.  Normally, data.obj should be a JSON object representing the call result according to our schema.  If you log "data" instead of "data.obj" then the problem is revealed:
swagger ready
{ url: 'https://api.eveonline.com/server/ServerStatus.xml.aspx',
  method: 'GET',
  headers:
   { 'transfer-encoding': 'chunked',
     'content-type': 'application/xml; charset=utf-8',
     'content-encoding': 'gzip',
     vary: 'Accept-Encoding',
     'access-control-allow-origin': '*',
     date: 'Fri, 06 Nov 2015 04:45:30 GMT',
     connection: 'close' },
  obj: null,
  status: 200,
  statusText: '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\r\n<eveapi version="2">\r\n  <currentTime>2015-11-06 04:45:31<
/currentTime>\r\n  <result>\r\n    <serverOpen>True</serverOpen>\r\n    <onlinePlayers>16503</onlinePlayers>\r\n  </resu
lt>\r\n  <cachedUntil>2015-11-06 04:45:44</cachedUntil>\r\n</eveapi>',
  data: '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\r\n<eveapi version="2">\r\n  <currentTime>2015-11-06 04:45:31</curre
ntTime>\r\n  <result>\r\n    <serverOpen>True</serverOpen>\r\n    <onlinePlayers>16503</onlinePlayers>\r\n  </result>\r\
n  <cachedUntil>2015-11-06 04:45:44</cachedUntil>\r\n</eveapi>' }
So we're getting the data, but it's being returned in the text field instead of being deserialized into a JSON object.  Hmmm...the docs for swagger-js seem to imply it can handle XML data.  They even give an example of how to request it.  But the smoking gun is this GitHub issue comment in swagger-codegen:

None of the Swagger clients support XML!
Well, that was a lot of time wasted.

Client is a Fail...Is This Hopeless?

Apparently none of the Swagger generated clients can handle XML data (I tried a Java client as well, same story).  Is there anything we can salvage out of this?  There are at least two positives we can take away:

  1. Swagger generates impressive documentation.  I'm not sure it's worth the time to write a complete Swagger spec just to get the documentation, but it's not entirely crazy as new tools are being written to translate Swagger into other API generator formats.  In fact, the Swagger-UI demo will provide online documentation for any spec with a URL behind it.  So if you're particularly lazy, you can expose your spec and just point people to the demo with your URL.
  2. You can add your own XML deserializer by either modifying one of the Swagger code generators, or deserializing after the fact.  The latter is particularly easy for Javascript clients as shown below.

It's an extra step, but it's pretty easy to convert XML to JSON in NodeJS using xml2js.  Here's what our script above would look like with this change:
var parseString = require('xml2js').parseString;
var client = require('swagger-client');
var swagger = new client({
  url: 'http://localhost/swagger.json',
  success: function() {
    console.log('swagger ready');
    swagger.Server.get_server_ServerStatus_xml_aspx({}, {}, function(data) {
      // Raw text to convert is in statusText
      parseString(data.statusText, function(err, result) {
        console.log(JSON.stringify(result));
      });
    });
  }
});
which produces the following output:
swagger ready
{"eveapi":{"$":{"version":"2"},"currentTime":["2015-11-06 12:48:45"],"result":[{"serverOpen":["True"],"onlinePlayers":["14711"]}],"cachedUntil":["2015-11-06 12:49:22"]}}
Xml2js has to make some guesses regarding types whether values are arrays, but this is something we can work with if needed.  Am I doing this in my code?  No.  I'd like a real client with properly structured response objects in many languages.  So my quest will continue.

Conclusion

I have to say I'm pretty bummed that Swagger doesn't support XML yet.  We could have knocked out a nice cross-language endpoint API with great documentation all in one fell swoop.  I looked around a bit for alternatives and found RAML (RESTful API Modeling Language).  I plan to test this out next and see where I get.

In the meantime, I'm still using Swagger for my own APIs and it should work fine for the EVE CREST endpoints.  That's a future project as well.

0 comments:

Post a Comment