Tuesday, April 26, 2016

EVE Online Static Data Export Web Service

As part of supporting the EVE Online third party community, CCP regularly releases the Static Data Export (SDE) which is a collection of database and YAML files containing static data used in the game.  This data provides reference information used by various third party tools.

The raw form of the data export is a bit unwieldy to use because it uses a mix of data formats which are convenient for managing versions and building in-game data, but less convenient for typical third party use which involves SQL style queries.  As a result, several third party conversions have been created that transform the SDE into various SQL vendor formats.

The database formats are easier to use, but you'll still need to set up a database and import the data.  The evekit-sde project is an attempt to go one better and provide a web service which exposes a REST API for access to all SDE data.  The REST API is provided by Swagger annotated endpoints.  This makes it very easy to view high quality documentation for the REST API, try out API calls directly, or generate clients in a variety of languages.  We'll demonstrate each of these features in this blog post.


EveKit SDE Public Site
GitHub Project Page

Viewing SDE Documentation and Trying out the API

Following the link to the public site (see above) will bring you to a landing page which lists all the current SDE conversions that are exposed by the site:

Clicking on an SDE release will bring you to the landing page for that release:

and, finally, clicking on "Swagger UI demo" will pull up the Swagger UI:

The Swagger UI demo page is hosted by the Swagger project, but can be used to view any Swagger configuration with a URL.  In this case, we insert the URL for the SDE service.

The main Swagger view shows all of the SDE sections exposed as web service endpoints.  Clicking on a section will show all the endpoints for that section.  There is one endpoint for each table in a given SDE section:

Selecting an endpoint will show a description of the section and includes a form for making a call to the endpoint:

Every endpoint call contains two (optional) parameters:

  • contid - this is a simple offset into the endpoint call result list
  • maxresults - the maximum number of results to return from the call.  The max is 1000.
These parameters are optional with sensible defaults.  All remaining parameters are called "selectors" and are used to filter the results returned from the call.  A selector is a JSON string with the following syntax:

  • {any: <boolean>} - Wildcard selector. If true, then this data field is not used to filter returned data. Setting this value to false has no effect.
  • {like: <string>} - String match selector. If the associated data field is string valued, then all returned data must satisfy the SQL expression 'field LIKE selector'. Normal SQL 'LIKE' syntax is allowed (e.g. % as wildcard).
  • {values: [<v1>,...,<vn>]} - Set selector. The associated data field of each returned data item must contain one of the listed values.
  • {start: <lower>, end: <upper>} - Range selector. The associated data field of each returned data item must satisfy lower <= value <= upper.

This syntax allows very flexible selection on SDE data. For example, to quickly find the inventory category for ships, set the categoryName selector to {like: "%Ship%" } on the inv/category endpoint:

which results in the response (click "Try it out!" to generate):

To find all types with "Raven" in the name, set typeName to {like: "%Raven%"} on the inv/type endpoint; and, to further restrict this set to those types actually on the market, set marketGroupID to {start: 0, end: 1000000}:

When you click "Try it out!" you'll see the set of Raven associated types on the market.

There are a few details you should be aware of when making calls:
  1. There is no explicit indication that there may be multiple pages of results.  To page through results, you'll need to call an end point until the number of returned results is less than "maxresults".  You'll also need to adjust "contid" for each call.  Typical code will look like this:

    contid = 0
    maxresults = 1000
    results = <call endpoint>(contid, maxresults, selectors...)
    while (results.length < maxresults) {
      ...Do something with data...
      contid += results.length
      results = <call endpoint>(contid, maxresults, selectors...)

  2. You can cut/paste the "Curl" line from the Swagger UI to call an end point from the command line.
  3. Every call has three possible results:
    • 200 (OK) will be returned when the call to the end point succeeds.
    • 400 (Bad Request) will be returned if a selector has an incorrect format.
    • 500 (Internal Server Error) will be returned if the end point service encounters an internal error.
  4. HTTP response headers are used to relay SDE service meta-data:
    • Date - server time when the result was returned (UTC)
    • Expires - the date when the returned data will expire. Defaults to 24 hours for SDE results (since these are unchanged between releases)
    • EveKit-SDE-Version - the name of the SDE release for which results were returned
  5. Calling the API from the Swagger UI doesn't show the Date and EveKit-SDE-Version response headers (but they are there).  If you use the curl command line and pass the -v option you can view the HTTP response headers directly.

Creating an SDE Client

The Swagger Editor makes it very easy to generate a client in your favorite language.  You can find instructions for this process in a previous post (see the examples section toward the end).  You'll need the URL for the Swagger description of the SDE version you want to build a client for.  This can be found on the landing page of the SDE version (see above).

Parting Words

The SDE service we've described here will likely work best for third party applications that only need select pieces of the SDE.  The query capabilities of selectors makes it easy to select just the data you need.  Larger, more complicated tools, will likely be better served by biting the bullet and importing the SDE dump as a database.


Post a Comment