Saturday, December 19, 2015

EVE Online XML API Endpoint Proxy

In an earlier post, I announced the release of the EVE XML API.  This is a Java library (at least the third or fourth by my count) which provides access to the EVE Online XML API server endpoints.  If you're writing third party code in Java, you might try this library out.  It's complete as of the latest release (not officially named -- but everyone is calling it the Frostline release).  It was also battle tested in a slightly different form as part of EveKit.

For small or one-off projects, it can be a pain to install a library.  Or, maybe you're using a language which doesn't have a ready made library for accessing the EVE Online XML API endpoints.  It's for these use cases that we've created the EVE XML API Proxy, a simple web service which exposes a REST API which in turn uses the EVE XML API library to call the EVE Online XML API server endpoints on your behalf.  In other words, a proxy for your EVE Online server requests.

To make all of this easier to use, we've annotated the proxy using Swagger.  This makes it very easy to view high quality documentation for the REST API, try out API calls directly, or generate proxy clients in a variety of languages.  We'll demonstrate each of these features in this blog post.


Live Proxy Site
GitHub Page
Swagger UI for Proxy
Swagger Editor/Code Generator (see instructions below)

A Brief Introduction to Swagger

I discussed Swagger in an earlier post, which I will briefly recap here.  Swagger is an API tooling ecosystem designed for REST based APIs.  It is open source with core libraries and tools available on GitHub (see the appropriate link under "Resources").

The Swagger tool chain is driven by an API specification written in a variant of JSON schema.  You can write a specification by hand, but it's also possible to auto-generate a specification by annotating an existing REST API.  We use the latter approach to generate a Swagger specification for the REST API exposed by our proxy.

There are three main tools which consume a Swagger specification:
  • Swagger Editor: This tool lets you create a Swagger specification online, test it out, then export a client or server back end in a variety of languages.  We use this tool below to show you how to generate a Python client for our proxy.
  • Swagger UI:  This tool turns a Swagger specification into online documentation.  The online documentation can also be used to try out the API (i.e. make API calls).  We use this tool next to introduce the proxy API and make a sample call.
  • Swagger Codegen: This is a build tool which converts a Swagger specification into a client or server back end in a variety of languages.  Much of the same functionality is built into the Swagger Editor, but the Codegen tool is usually more up to date and supports more languages.  Note that code generation for Javascript is usually not done statically.  Instead, you use the swagger-js Javascript module which generates a Javascript API dynamically from a Swagger specification.  We use the swagger-js module below.  We won't be using Swagger Codegen in this writeup.
Swagger has an active developer community and good support for many languages.  Unfortunately, it doesn't support XML very well, which is why I failed to create a direct Swagger specification for the EVE Online XML endpoints.  It's also worth noting that Swagger has competition in the form of RAML (RESTful API Modeling Language).  RAML has some nice features and better support for XML, but a much less active developer community with fewer languages supported.  I also ran into problems creating a direct specification for the EVE Online XML endpoints using RAML (I haven't written up that experiment yet).

Anyway, back to the proxy...

Viewing Documentation and Trying out the API

Swagger makes it very easy to view high quality documentation for REST APIs.  The easiest way to view the documentation for the proxy is to use the Swagger UI online demo pointed at our Swagger specification.  This link will do that.  If you click the link, you'll see a view like this:

This view shows each of the API sections exported by the proxy.  Clicking on the Server section, and then again on the first entry, will produce a detailed view of the "ServerStatus" endpoint:
This view gives a detailed description of the endpoint including a description of the expected return values.  In this case, there are three possible return codes: 200 (OK), 404 (server error), or 500 (service error).  The model schema is shown for each response type.

You can try calling this endpoint by clicking the "Try it out!" button, which will give a display like the following:
Each part of the response is shown with the content sent back from the server.

This example illustrates a few features worth discussing more carefully:
  1. You can cut/paste the "Curl" line to call the proxy end point from the command line.
  2. Every call has three possible results:
    • 200 (OK) is always the result when the call to the EVE Online XML server succeeds.
    • 404 (Not Found) is always the result when the call to the EVE Online XML server succeeds, but EVE returns one of the error codes defined in the error list.  In this case, the result body contains the error code, message and time fields.
    • 500 (Internal Server Error) is always the result when the proxy service itself has an error, either because there was an IO error attempting to reach the EVE Online XML server, or there was an internal error in the service.  In this case, the result will contain a simple error message.
  3. HTTP headers are used to relay certain EVE Online XML server fields
    • The "Date" HTTP field contains the "currentTime" field returned by EVE
    • The "Expires" HTTP field contains the "cachedUntil" field returned by EVE
    • The "EVEXML-Version" custom HTTP field contains the EVE API version parameter returned by EVE
  4. Calling the API from the Swagger UI doesn't show the Date and EVEXML-Version API headers (but they are there).  If you use the curl command line and pass the -v option you can view the HTTP headers directly.
You may have noticed an optional parameter for this endpoint called "server".  This optional parameter is available on every endpoint and provides a way to tell the proxy to call a server other than the default production EVE Online XML server (e.g.  For example, if you want to invoke the test server you can enter "" in the server field and get a result like the following:
Endpoints which require credentials will have parameters for passing api keys and vCodes.  For example, the CharacterSheet endpoint:

Example: Using the Proxy from Javascript

We've seen how to view proxy documentation and try out a few calls manually.  Now let's look at how you can use the proxy from within a web page using Javascript.  This is a simple example, but it's worth doing because Swagger provides a dynamic generator for Javascript clients.  This is different than generating clients for other languages which is mostly done statically (as shown in the Python example below).

The following HTML document illustrates how to retrieve and display server status (our GitHub page contains similar documentation):

<!DOCTYPE html>
  ServerStatus Call Example Using EVE XML API Proxy

  <div id="mydata"></div>
  <script src='' type='text/javascript'></script>
  <script type="text/javascript">
    var url = "";
    window.swagger = new SwaggerClient({
    url: url,
    success: function() {
    // On success, fetch server status and display it in the DIV above
    swagger.Server.requestServerStatus({}, {}, function(data) { document.getElementById("mydata").innerHTML = JSON.stringify(data.obj); });

Note the use of to pull in the Swagger javascript module.  This is necessary because GitHub doesn't attach proper content types to raw files.  Rawgit does this for us so that we can pull the raw module directly into our code.

Loading this page will product output like the following:
The format of a Javascript library call is:
swagger.<api_section>.<method>({<method args>}. {<header args>}, success_callback, failure_callback) 
where "api_section" is one of the tags defined in the Swagger specification (e.g. Server, Character, Corporation, etc.) and "method" is an operation ID defined in the specification.  Unfortunately, you have to look at the Swagger specification to determine operation ID as neither the Swagger UI nor the Swagger Editor show this information in the documentation they generate.  To make it easier to use the proxy, we've consistently set the operation ID to be "request" plus the name of the XML end point (e.g. requestServerStatus).

Method arguments are set in the first argument to the request method.  For example, the following call sets the optional "server" parameter in the server status request:
swagger.Server.requestServerStatus({server: ""}, ...)
Header arguments can be set in the second argument to the request method.  You can use a header argument to do things like specify the content type of the response.  We don't use header arguments for the proxy (we only support one response type).

Finally, all method calls are asynchronous following the usual Javascript convention.  The result of a call is an object with several useful fields:
  • status: the HTTP status code for the response.
  • headers: an object containing the HTTP headers returned on the response.  This is where you'll find the "expires" header describing the expiry time of the EVE server result.
  • obj: a Javascript object representation of the result.  The format of this object is determined by the schema specified in the Swagger specification.
  • data: a JSON encoded string representation of the result.
  • url: the URL to which the request was made.
In the example above, we convert "obj" to a string for display on the web page.

Example: Using the Proxy from Python

Now let's look at an example where we generate and use a Python client.  In this case, we'll need to generate the client using the Swagger Editor.  By default, the editor loads with a sample specification.  To load the specification for the proxy, select "File -> Import URL..." and enter "":
Click "Import" then wait a few seconds for the editor to process the specification.

The left side of the editor will show the specification in YAML format.  The right side of the editor shows a view similar to the Swagger UI, including the ability to try the API directly.  Code generators are selected from the menu at the top.  For this example, we'll select "Generate Client -> Python".  This will download the Python client code.

The client download will consist of a zip file containing a single directory called "python-client".  Within this directory there is a and a  The setup file will install the library as "swagger_client" including any needed dependencies.  The README file gives sample code for using the library (note: the instructions in the README file do not instantiate ApiClient correctly.  See the example below for the proper way to create an ApiClient).

I'm not going to install the library for this example.  Instead, I'm going to just load and use the library directly from the "swagger_client" folder.  To do that, we need to make sure we have all the dependencies listed in  In my case, I was missing urllib3 so I installed that.  Once you've installed all necessary libraries, it is straightforward to make calls to the API:

>>> # The local directory is "python-client"
>>> import swagger_client
>>> api_client = swagger_client.ApiClient()
>>> server_api = swagger_client.ServerApi(api_client)
>>> server_api.request_server_status()
{'online_players': 20845, 'server_open': True}

The call format is similar to the Javascript example except that the method name consists of "request_" followed by the name of the end point converted to lowercase and separated on word boundaries with "_".

By default, API calls are synchronous (this is shown in the example above).  To make an asynchronous call, define a callback function and pass it to the method call:

>>> def cb(obj):
...  print(obj)
>>> server_api.request_server_status(callback=cb)
<Thread(Thread-4, started 9632)>
>>> {'online_players': 22299, 'server_open': True}

Method arguments are passed as named arguments.  For example, to use the test server:

>>> server_api.request_server_status(server="")

The result of a call is a typed Python object matching the schema defined in the Swagger specification.  You can find Python class definitions for these objects in the swagger_client/models directory

Parting Words

The proxy we've described here uses Swagger annotations to provide nice documentation and a convenient interface for ad hoc use of the EVE Online XML end points.  If you already have a favorite third party library, then this proxy may not be for you.  But if you're looking for a library, or if you're looking to use a language that isn't commonly supported, then our proxy with Swagger generated clients might be right for you.

Thursday, December 10, 2015

EVE XML API: Yet Another EVE XML Server Library (in Java)

The world probably doesn't need another library for accessing the EVE XML server endpoints, but I promised to open source the code for EveKit and this is the first bit I've had time to clean up and release.  If you don't know what EveKit is, keep watching this blog especially in the Spring of 2016 as I plan to relaunch an updated (and hopefully improved) EveKit.

  • This library is open source and available on GitHub.  The README provides a simple quick start for the impatient.
  • The library builds with Maven and has been published to Maven Central.  If you're using Maven, then it's easiest to just add the following dependency:
  • The javadocs are not great, but the API is also very straightforward and follows the EVE XML endpoint format pretty closely.  Eventually, I'll get around to filling out proper javadoc. 
Please feel free to comment here or raise issues on the GitHub project to report bugs, request features, etc.

Saturday, November 14, 2015

Swagger for the EVE XML API? Not so much...

There are plenty of third party libraries for accessing the EVE XML API in a variety of languages, some of which I have (or will) reviewed on this blog.  That said, each library has its own idiosyncrasies and each implementation takes a slightly different view of how to handle errors, package results, etc.  Each library also differs in the level of documentation provided, with the default usually being none.

Wouldn't it be nice to have a way to describe the EVE XML API in some API description language, then have a tool which generates bindings in several different languages?  Well yes, and such things have existed in computer science for many years (anyone else remember CORBA IDL?).  A recently popular library for doing this is Swagger, which is designed to model REST APIs using an extension of the JSON schema specification.  Swagger has an impressive set of tools for online editing of a REST API, trying out an API out against a live site, and generating clients (and servers!) in a variety of languages.  Also, they seem to claim in various places that XML is supported.  This sounds exactly like the right thing to try for the EVE XML API.  Except it doesn't work.  The rest of this blog talks about my attempt to make this work and ultimately what failed (short version: none of the generated clients support de-serializing XML documents yet).


Swagger Site
Sample Swagger spec for the EVE XML API

A Quick Intro to Swagger

In a word, Swagger is an API tooling ecosystem designed for REST based APIs.  Swagger is open source with the core libraries and tools all available on GitHub.  A company called SmartBear (of Code Collaborator fame) is behind much of the Swagger code, but there are many open source contributors as well.

Swagger takes as input an API spec written in a variant of JSON schema.  The spec itself can be represented as either YAML or JSON (these are largely interchangeable).  There are three main tools which consume Swagger API specs:

  • Swagger Editor: This tool lets you create an API spec online, test it out, then export a client or server backend in a variety of languages.  We use this tool below to create our EVE XML API spec.
  • Swagger UI: This tool turns an API spec into online documentation.  There's an online demo which lets you pull up any publicly available Swagger spec.
  • Swagger Codegen: This tool converts an API spec into a client or server backend in a variety of languages.  The Swagger Editor already has this functionality built in, so normally you'd only use the code generator directly if you need more control over the process, or if you plan to write your own generator.  If you plan to use your API mainly from the browser or other Javascript environment, then you don't even need to export a custom client.  Instead, you can use swagger-js which reads any API spec and provides the appropriate calls.  We show how to use this below as well.

The online tooling and capabilities of Swagger look very impressive.  This would seem to be a great way to document and provide endpoints for the EVE XML API.  Let's see how that goes...

Let's Make a Model for the EVE XML API

To get started, let's try making a model for the EVE XML API.  The model will define all the operations we can invoke, the arguments for those operations, and the structure of the return values.  We'll use the Swagger Editor to create and test our spec.  You can download the editor and run it locally, or you can also just run the live demo which is usually easier.  Here's what the editor looks like:

Swagger Editor Live Demo
On the left is a text editor where you create your API spec, and on the right is a live rendering of your spec including tools to try out your API calls.  There are numerous examples you can view from the File...Open Examples menu.  The menu also contains drop downs for generating clients and server back-ends in various languages.

To start out, we'll implement the ServerStatus API call.  This is a very simple call which takes no arguments and requires no API key.  We'll start with a clean slate, then build up the API file.  So we'll select File...New in the Swagger Editor to start with a basic template.  If you want to skip ahead a bit, you can just download the swagger.yaml file directly.  Or keep reading to understand how we're building this file from scratch.

The first part of the configuration is preamble describing the API, giving its location, the schemes we can use to access the API, and the type of data the API will produce:
swagger: '2.0'
  title: Eve Online XML Endpoint API
  description: Eve Online XML Endpoint API
  version: 1.0.0
  - https
  - application/xml
Most of these settings can be overridden for each API call.  This isn't necessary for the EVE XML API since every call has the same basic properties.  So in this case the preamble says that all calls will hit using the https scheme, and every call is expected to produce xml output.

Next, we need to describe the actual API endpoints.  These are called "paths".  In our example, we're calling ServerStatus, so the complete path specification will look something like this:
      summary: Current Tranquility server status
        - Server
          description: Server status
          description: Error
The "paths" keyword marks the start of the paths section and is followed by entries which give the actual path, the HTTP methods which may be invoked on that path, and the responses to expect.  In this example, the path is "/server/ServerStatus.xml.aspx" which is appended to the host (in the preamble) to give the complete path of this call.  We call this path with the "get" HTTP method, and expect two basic responses: 200 (OK) or 400 (Bad Request).  These responses are specified in the "responses" section.  The "tags" keyword is used to group this call for client generation and documentation purposes (we'll show that further below).

At this point, we actually have a usable spec which you can use to call the API.  If you're following along in the editor you'll see something like the following in the right panel:
Call Test Panel
If you click on "Try this operation" and hit "Send Request" (and you're connected to a network) then you should see a successful API call (click on "Pretty" or "Raw" in the output panel to see the XML output).

This is nice, but not much use without something which parses the response and gives us a nice structured object.  Swagger lets you do this by defining a schema for each response.  For the ServerStatus call, there are two types of response we need to handle.  A success response will look like this:
<?xml version='1.0' encoding='UTF-8'?>
<eveapi version="2">
  <currentTime>2015-11-05 23:22:36</currentTime>
  <cachedUntil>2015-11-05 23:24:19</cachedUntil>
It's also possible to receive an error response, which will have this form:
<?xml version='1.0' encoding='UTF-8'?>
<eveapi version="2">
  <currentTime>2015-11-05 13:16:33</currentTime>
  <error code="106">
  Must provide userID or keyID parameter for authentication.
  <cachedUntil>2015-11-06 01:16:33</cachedUntil>
Note the common elements in both responses.  Ideally we'd like our schema to reflect these common elements as well.  Here's one way to specify such a schema in Swagger:
    type: object
      name: eveapi
        type: integer
          attribute: true
        type: string
        type: string
      - $ref: '#/definitions/ServerResponse'
      - type: object
            type: string
                type: integer
                  attribute: true
      - $ref: '#/definitions/ServerResponse'
      - type: object
            type: object
                type: boolean
                type: integer
Swagger schemas live in the "definitions" section of the specification.  We first define a ServerResponse type which captures the common elements of all responses (version, currentTime and cachedUntil).  XML definitions in Swagger require a bit more work because XML allows both attributes and values.  In the case of the ServerResponse, we use the "xml" tag to indicate certain XML features.  The "name" tag defines the name of the root tag for the ServerResponse (i.e. eveapi).  The "attribute" tag indicates which properties are XML attributes.  One more detail: the Swagger date-time type can't parse EVE XML API time values, so we have to use "string" for those values instead.

We'll use the ServerResponse schema as a base definition for two additional schemas: ErrorResponse and ServerStatus.  The Swagger tag "allOf" is a schema composition operator.  This operator accepts a list of schema object definitions which are combined to form a single schema.  For example, the ServerStatus schema specification says to combine all of the fields of ServerResponse with the schema which defines a "result" property, and two additional properties for serverOpen and onlinePlayers.  Note the use of the $ref operator to pull in the definition of the ServerResponse schema.

The last step is to add the appropriate schemas to the response section for the ServerStatus call.  The new "paths" section now looks as follows:
      summary: Current Tranquility server status
        - Server
          description: Server status
            $ref: '#/definitions/ServerStatus'
          description: Error
            $ref: '#/definitions/ErrorResponse'
Definitions done, let's see if we can make this into a real XML API client.

Nice Model...Let's Make a Client!

If you're using the Swagger Editor with our example, then at this point you have a usable spec which you can try out.  However, try as I might, I could never get the Swagger Editor to render the response objects (as displayed in the "Rendered" tab of the response when you try the API).  That was my first clue that maybe this wasn't going to work so well.

I decided to press on and generate a client.  Maybe it's just the Swagger Editor that is broken, right?  The easiest client to generate is a Javascript client.  This is easy to do because the swagger-js project does it for you on the fly.  There are a few different ways to test this out.  I decided to use the NodeJS approach as documented on the site.  The following code snippet does the trick:
var client = require('swagger-client');
var swagger = new client({
  url: 'http://localhost/swagger.json',
  success: function() {
    console.log('swagger ready');
    swagger.Server.get_server_ServerStatus_xml_aspx({}, {}, function(data) {
There's one catch here in that the Javascript client wants to load your Swagger spec from a url.  To make that work I used my locally installed web server and put my spec there.  The nice trick with the swagger-js client is that the same code works for any valid Swagger spec.  So there isn't a separate generation step, you just drop this code into your app and you're good to go.

If you run this, however, you won't get what you expect because "data.obj" will be null.  Normally, data.obj should be a JSON object representing the call result according to our schema.  If you log "data" instead of "data.obj" then the problem is revealed:
swagger ready
{ url: '',
  method: 'GET',
   { 'transfer-encoding': 'chunked',
     'content-type': 'application/xml; charset=utf-8',
     'content-encoding': 'gzip',
     vary: 'Accept-Encoding',
     'access-control-allow-origin': '*',
     date: 'Fri, 06 Nov 2015 04:45:30 GMT',
     connection: 'close' },
  obj: null,
  status: 200,
  statusText: '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\r\n<eveapi version="2">\r\n  <currentTime>2015-11-06 04:45:31<
/currentTime>\r\n  <result>\r\n    <serverOpen>True</serverOpen>\r\n    <onlinePlayers>16503</onlinePlayers>\r\n  </resu
lt>\r\n  <cachedUntil>2015-11-06 04:45:44</cachedUntil>\r\n</eveapi>',
  data: '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\r\n<eveapi version="2">\r\n  <currentTime>2015-11-06 04:45:31</curre
ntTime>\r\n  <result>\r\n    <serverOpen>True</serverOpen>\r\n    <onlinePlayers>16503</onlinePlayers>\r\n  </result>\r\
n  <cachedUntil>2015-11-06 04:45:44</cachedUntil>\r\n</eveapi>' }
So we're getting the data, but it's being returned in the text field instead of being deserialized into a JSON object.  Hmmm...the docs for swagger-js seem to imply it can handle XML data.  They even give an example of how to request it.  But the smoking gun is this GitHub issue comment in swagger-codegen:

None of the Swagger clients support XML!
Well, that was a lot of time wasted.

Client is a Fail...Is This Hopeless?

Apparently none of the Swagger generated clients can handle XML data (I tried a Java client as well, same story).  Is there anything we can salvage out of this?  There are at least two positives we can take away:

  1. Swagger generates impressive documentation.  I'm not sure it's worth the time to write a complete Swagger spec just to get the documentation, but it's not entirely crazy as new tools are being written to translate Swagger into other API generator formats.  In fact, the Swagger-UI demo will provide online documentation for any spec with a URL behind it.  So if you're particularly lazy, you can expose your spec and just point people to the demo with your URL.
  2. You can add your own XML deserializer by either modifying one of the Swagger code generators, or deserializing after the fact.  The latter is particularly easy for Javascript clients as shown below.

It's an extra step, but it's pretty easy to convert XML to JSON in NodeJS using xml2js.  Here's what our script above would look like with this change:
var parseString = require('xml2js').parseString;
var client = require('swagger-client');
var swagger = new client({
  url: 'http://localhost/swagger.json',
  success: function() {
    console.log('swagger ready');
    swagger.Server.get_server_ServerStatus_xml_aspx({}, {}, function(data) {
      // Raw text to convert is in statusText
      parseString(data.statusText, function(err, result) {
which produces the following output:
swagger ready
{"eveapi":{"$":{"version":"2"},"currentTime":["2015-11-06 12:48:45"],"result":[{"serverOpen":["True"],"onlinePlayers":["14711"]}],"cachedUntil":["2015-11-06 12:49:22"]}}
Xml2js has to make some guesses regarding types whether values are arrays, but this is something we can work with if needed.  Am I doing this in my code?  No.  I'd like a real client with properly structured response objects in many languages.  So my quest will continue.


I have to say I'm pretty bummed that Swagger doesn't support XML yet.  We could have knocked out a nice cross-language endpoint API with great documentation all in one fell swoop.  I looked around a bit for alternatives and found RAML (RESTful API Modeling Language).  I plan to test this out next and see where I get.

In the meantime, I'm still using Swagger for my own APIs and it should work fine for the EVE CREST endpoints.  That's a future project as well.

Wednesday, November 11, 2015

Welcome - Here's what this blog is about...

So...what is this?

This blog is about third party development for EVE Online, a sandbox space game set in the very far future.  EVE Online is unusual among MMOs in that a rich API is available for collecting in game data.  Over time CCP, the company behind EVE Online, is adding APIs to manipulate data outside of the game as well.  Many other MMOs have since started adding similar capabilities, but EVE Online was one of the first, and has the most capability (as far as I know).

On this blog, I plan to do two things:
  1. Provide regular reviews of third party tools.  There are many.  I'll need help to cover them all, and I'll be asking for help over time.
  2. Provide guidance for developers or others interested in getting into third party development on EVE Online.  There is an active development community, but not a great collection of how to's or other instruction.  Documentation is getting better, but also somewhat incomplete.  I'll try to plug some of those holes in this blog.
I plan to add a fair amount of content through video reviews and tutorials, supplemented with examples and occasional write-ups of material which makes better sense in written form.

And you are?

I'm a professional software developer and a terrible EVE player.  These days, I mostly play EVE solo.  I keep a few accounts around and I have a small one-person EVE corporation which allows me to experiment with more parts of the EVE API.  I'm also the creator of EveKit which recently shutdown due to lack of interest (maybe I'll re-launch, I've been missing it lately).

I've posted a few videos before of a tutorial nature.  I also gave a talk about third party development at FanFest 2015 in Iceland.  Finally, I'm a contributor to the EVE Online API documentation site.  Suffice it to say, I try to stay active in the Eve third party development community.

What can we expect next?

One of my first content posts will be a basic orientation to third party development for EVE Online.  I'll be including lots of links to important resources, and I'll call out some of the more popular third party libraries.  Don't be offended if I miss your favorite library.  All of these will go into my matrix of third party tools and apps, and there will be a way to suggest other tools or apps I should cover.