Friday, December 23, 2016

SDE 2016-12-13 released

We apologize for the delay in getting the latest SDE out (holidays, etc).

We've released the 20161213 SDE at  You can access the SDE directly through that link, or go to the SDE section at

If you're using the "latest" alias ( then you'll automatically be using the new SDE.

Thursday, December 8, 2016

Human readable date fields added for convenience

We performed a small maintenance release this morning to add "date" fields to model elements to make it easier for humans to read raw data.  Every field which records a timestamp (e.g. lifeStart, sentDate, etc.) is now replicated in a separate field with the suffix "Date" attached.  This new field records the timestamp as a date if it is valid, and null otherwise.  For example, suppose a model object has the field:
lifeStart: 1457616630360
Then a new field will be added to this object with value:
lifeStartDate: "2016-03-10T01:30:30.360Z"
The representation of this field depends on the client you are using.  Javascript will report this field as a String, for example, which can be parsed into a javascript Date if desired.

We've made this change to every model object with a timestamp field, including reference data.

Saturday, December 3, 2016

Maintenance Complete - 2016-12-03

We've completed the upgrade of EveKit with the following changes and bug fixes:

  • New Feature: we've started synchronized the "reference" data provided by the XML API servers.  These are the "server status", "API calls", "EVE" and "Map" endpoints.  As with all other synchronized data, we'll be tracking historical changes to reference data.  You can browse reference data from the main page under the API dropdown.  Use the provided Swagger URI to create a swagger client for accessing reference data.
  • Improvement: for character mail messages, we've renamed the "corpOrAllianceID" to "toCorpOrAllianceID" to align better with the XML API.
  • Bug Fix: we've added the "ownerTypeID" field to upcoming calendar events.  This field was previously missing.
  • Bug Fix: we've added the "owner1TypeID" and "owner2TypeID" fields to the wallet journal.  These fields were previously missing.
  • Improvement: we've upgraded the backing database to UTF multi-byte in order to properly capture UTF characters.  This means emojis will now be captured properly in the backing database.
  • Bug Fix: all fields representing time stamps will be uniformly set to -1 (an illegal UNIX time) when the corresponding field is unset in the XML API.  For example, an unfinished industry job will have a completion time of -1 until the job completes.

Maintenance starting - 2016-12-03

Maintenance has started.  Synchronized accounts will cease to update for the duration of the maintenance.  We expect to complete the maintenance at 04:00 UTC.

Thursday, December 1, 2016

Downtime for Site Update on 2016-12-03

We'll be pausing account synchronization on 2016-12-03 for one hour starting at 03:00 UTC and ending no later than 04:00 UTC.  During this time, we will be releasing version 3.0 of EveKit which includes several bug fixes as well as a new data set consisting of XML API reference data (server status, API calls, EVE reference data, and map reference data).  We synchronize reference data in the same way we synchronize account data, meaning you'll be able to access historical versions of reference data if needed.  Reference data will not require an access key.  We will post more information about reference data at update time.

We will post to the site when we start updates.  During this time, accounts will not synchronize but will be accessible through the usual APIs.  You may observe minor data discrepancies as we update the database.  Account synchronization will resume at the end of the update period, which we will also post to the site.

Thursday, November 17, 2016

SDE 2016-11-14 released

We've released the 20161114 SDE at  You can access the SDE directly through that link, or go to the SDE section at

If you're using the "latest" alias ( then you'll automatically be using the new SDE.

Friday, October 21, 2016

Gaps today due to DNS attack

The well publicized DDoS DNS attack today caused gaps in EveKit synchronization and market data coverage.  The first attack started approximately 11:00 UTC and ended just after 13:00 UTC.  A second prolonged attack started just before 17:00 UTC and ended around 21:00 UTC.  During these attacks many account sync requests failed due to the inability to resolve the location of some EVE services.  You'll see these outages in your sync history as red blocks, e.g.

Market data downloads were also affected.  During the attacks, book data was not able to update and will show the state of the book just before the attack began.  Market history (daily summaries) should not be affected.

Tuesday, October 18, 2016

SDE 2016-10-11 released

We've released the 20161011 SDE at  You can access the SDE directly through that link, or go to the SDE section at

As part of this release, we've also enabled an alias "latest" which always points to the latest Swagger config file for the SDE.  Starting with this release, you can use this link: to refer to the latest Swagger config.  If you use this link in your client generation code, then you'll always generate the latest client.  In most cases this won't be a problem.  We'll post here if a release includes a backwards incompatible change.

Tuesday, October 11, 2016

Two small bug fixes just released

We've just released two quick bug fixes:

  • Member tracking wasn't passing the "extended" flag correctly causing EveKit to only sync the limited version of this XML API.  We've fixed this so that sync requests from here on should retrieve extended member tracking information if your XML API key allows it.
  • Wallet journal sync incorrectly assumed that refIDs were unique across ALL wallet divisions.  This isn't true for transfers between wallet divisions which use the same refID to record the transfer in each division*.  In the past, EveKit would have recorded only one of the two journal entries leaving a gap.  Going forward, both entries will now be properly recorded.
*Shout out to Golden Gnu for pointing out this bug.

Sunday, October 9, 2016

Flat asset list downloads

We've just released a small update to EveKit which will sync account assets using both the nested and flat AssetList API calls.  The way this works is as follows:

  1. We first call the nested AssetList API method.  This method indicates containment relationships between assets which we preserve in EveKit.
  2. We then call the flat AssetList API method and add any assets which were not returned in the nested call.  These assets will not have containment information as that information is excluded in the flat call.
  3. We finish by sync'ing the downloaded asset list with existing assets stored in your account, e.g. end of life assets you no longer possess, update any containment relationships which have changed, etc.
We added the call to the flat AssetList API in order to capture assets stored in Citadels, which are currently excluded from the nested call.  The flat API endpoint also includes some unexpected items, such as skills you have trained, and installed implants.  We leave it to the EveKit user to filter out these assets if they are not important.

Saturday, October 8, 2016

Market History Delay - 2016-10-07

There was a problem with the market history uploader last night which corrupted the last several days of market history for several market types.  Order book data was not affected.  The corrupted files have been fixed, but data for 2016-10-06 will be missing until the uploader runs again tonight.  Tomorrow morning (2016-10-09) at around 09:00 UTC the gaps for 2016-10-06 and 2016-10-07 will be filled.

We apologize for the outage.

Monday, October 3, 2016

EveKit 2.1.0 release

We've just released version 2.1.0 of EveKit.  This release fixes a number of type errors in sync'd data, including missing fields for some object types.  The second big change in this release is support for the "locations" XML endpoint.  If your registered EVE XML API key allows access to assets and locations, then EveKit will request the location of each of your assets.  These locations are stored with complete history as with any other EveKit object.

A new "locations" endpoint has been added to the EveKit API to provide access to the new data.  You'll need enable access to locations in your EveKit API access keys to access the new data.

The following type errors and missing fields were corrected:

  • AccountStatus.logonCount from int to long
  • AccountStatus.logonMinutes from int to long
  • APICall.groupID from int to long
  • CalendarEventAttendee.eventID from int to long
  • CharacterSheet.freeSkillPoints from int to long
  • SkillInfo.freeSkillPoints from int to long
  • UpcomingCalendarEvent.eventID from int to long
  • ContainerLog.itemTypeID from long to int
  • ContainerLog.locationID from int to long
  • ContainerLog.quantity from int to long
  • ContainerLog.typeID from long to int
  • MemberTracking.baseID from int to long
  • MemberTracking.locationID from int to long
  • Outpost.solarSystemID from long to int
  • Starbase.locationID from int to long
  • Asset.quantity from int to long
  • Asset.rawQuantity from int to long
  • Contract.startStationID from int to long
  • Contract.endStationID from int to long
  • Contract.volume from long to double
  • ContractItem.rawQuantity from int to long
  • IndustryJob.activityID from long to int
  • IndustryJob.blueprintTypeID from long to int
  • IndustryJob.productTypeID from long to int
  • IndustryJob.timeInSeconds from long to int
  • IndustryJob.solarSystemID from long to int
  • WalletTransaction.typeID from long to int
  • WalletTransacton.clientTypeID missing (int)
  • WalletTransaction.characterID missing (long)
  • WalletTransaction.characterName missing (String)

Saturday, September 17, 2016

SDE 2016-09-13 released

We've released the 20160913 SDE at  You can access the SDE directly through that link, or go to the SDE section at

Thursday, August 11, 2016

New SDE released and in new format

We've released the 20160809 SDE at  You can access the SDE directly through that link, or go to the SDE section at

For this release (and the 20160704 release as well), we've switched to using Fuzzy Steve's SDE dumps.  We've had to do this because Desmont McCallock decided to retire from EVE which means his conversion tool is now no longer maintained (and is now broken with the re-work of traits in types.yaml).  That tool is written in C# which unfortunately we can't support, so we've switched to Steve's dumps.  We had originally used Desmont's converter because it was open source and therefore we had more control over resolving issues as the SDE changes.  Fortunately, Steve has now open sourced his converter as well.  That gives us the control we need and moreover Steve's tool is written in Python which we'll be able to support should Steve stop providing his dumps.

Saturday, July 9, 2016

Market Data Outage - 2016-07-09 00:00 UTC to 2016-07-09 11:20 UTC

Many of you already know that CREST went down last night.  It appears that CREST returned around 10:30 UTC but the EveKit MarketData scraper did not automatically recover (it was supposed to, not clear yet why it didn't work).  The scraper was restarted at around 11:20 UTC.  There will be a gap in book data for all regions during this timeframe.  Market history data should not be affected.  The EveKit MarketData Service front end was not affected, other than not having recent data to show during the outage.

Monday, July 4, 2016

A site refresh including new Market Data endpoints

We released a refresh of the main EveKit site today.  The quick summary is:

  • The (now) four main offerings of the site are now accessible from the front page.  They are:
    • Account Synchronization - the original purpose of EveKit.
    • XML API Proxy -  a simple proxy to the EVE Online XML API endpoints.
    • SDE - Info and links to the online SDE versions that we host at EveKit.
    • MarketData - NEW! Info and links to the online EVE MarketData we host.
  • There's a nice little navigation bar in the top left corner that makes it easy to switch between offerings.
  • The new exciting feature is market data.  Some brief notes about that below.


EVE Online Market Data Access

About a year ago CCP exposed CREST endpoints for retrieving market data for EVE Online.  Recently, they've made many improvements to these endpoints, most notably providing the ability to download complete market data for a single region in one bulk download (technically, this bulk download would consist of several pages for a non-trivial region, but you get the idea).

Here at Orbital Enterprises, where our mantra is essentially "keep history for everything", we decided to start capturing this data for possible future use in EVE market play (region arbitrage, market cornering, station trading, etc).  To do that, we created a market data scraper which runs 24/7 with a daily archive run to package and store scraped data into a more convenient format.  We've built our scraper on top of our existing EveKit stack, the result of which is the EveKit MarketData Project which you can download from GitHub and try for yourself.

Scraping market data is good but one problem is that archive data and live data are in two different places (and two different formats): archive data is stored in Google Cloud Storage (see the EveKit MarketData Overview for more info) in a format convenient for certain types of extraction; but the most recent data is on the server in a snapshot format waiting to be archived at the end of the day.  This is kind of a pain if you're trying to develop an online tool which uses market data.  So we built a simple market data server which provides per-type, per-region access to either order books or market history on any given date we support (for book data, back to 2016-05-31; for market history, back to 2015-04-01).  The result of that effort is the EveKit MarketData Server Project which you can try out live here.

As a quick demonstration, we built a simple market viewer using our market data server.  You can try out the viewer here.  The code for the viewer is provided in the EveKit Frontend.

Using Market Data Offline

We'll be publishing an article or two on how to use EVE market data archives for developing trading strategies in the near future.  Those articles will be based on using Python and Jupyter notebooks.  In the meantime, the best documentation for how to retrieve and use EveKit market data archives offline is on the EveKit site itself here.

Saturday, June 18, 2016

Ridiculously Easy Autocomplete with the EveKit Online SDE

If you don't know already, here at Orbital Enterprises we host the two most recent versions of the EVE Online Static Data Export as a web service.  I posted about this previously here.

As I mention in the original blog post, the online SDE is mostly useful for short queries or lookups, and isn't the best thing for complicated queries (e.g. things that would end up being complicated joins if you had direct access to the database).  However, for web developers out there, it turns out to be ridiculously easy to implement nice features like EVE type name auto-completion using the online SDE.  I'll show you how to do that using javascript in this blog entry.


EveKit SDE Public Site
GitHub Project Page
Sample Code

Annotated Example

You can download the sample file (auto.html) from the link above.  You should be able to point your browser to a local copy of the sample and auto-complete EVE type names in the text box.  We'll go over the important parts of the sample file below.

We start out by loading a few important libraries in the header:

<!doctype html>
<html lang="en">
  <meta charset="utf-8">
  <title>autocomplete demo</title>
  <link rel="stylesheet" href="">
  <script src=""></script>
  <script src=""></script>
  <script src='' type='text/javascript'></script>

We use jQuery's auto-completion support to drive our flow.  You don't have to use jQuery to do this; there are other javascript libraries that support auto-completion.  In fact, all you really need is something that will accept text and periodically query an auto-completion routine to return possible choices.  But jQuery is one of the easiest to use, so that's what we use here.

We also load the Swagger javascript client library.  This library will turn the online SDE API spec into a usable javascript client.  We talk about Swagger all the time with EveKit because all our APIs are annotated with Swagger, making it very easy to build clients for our tools.  See some of our other posts for more detail on this topic.

There are two important bits of code in the sample file.  The first is the code which builds the Swagger client:

var url = "";
var swagger = new SwaggerClient({
    url: url,
    usePromise: true
}).then(function(obj) {
    // save a debug copy of swagger as well as returning
    window.swagger = obj;
    console.log("swagger ready");
    $( "#typecomplete" ).autocomplete({ source: typeRequester });
    return obj;
}).catch(function(error) {
    console.error('Swagger promise rejected', error);
    return null;

The SwaggerClient class comes from the Swagger javascript library and is parameterized with a URL pointing to the location of the online SDE API specification.  In this example, we're using the most recent SDE at time of writing.  This will change over time, so be sure to visit the SDE page on a regular basis to get the latest URL.

The calling style used here is based on "promises" (and is initiated by passing the "usePromise: true" flag to Swagger).  Promises are a very clean approach to handling the asynchronous nature of remote calls in javascript.  You can read a very accessible introduction to promises here.  But even if you've never seen promises before, it's easy to intuit how the code works here: first we create the SwaggerClient object with an appropriate URL; THEN we save the result of that call and initialize the type completer; if something goes wrong, we CATCH the error and do something appropriate.

So the net result of this call is that we'll have an initialized text box and a Swagger client ready to retrieve data from the online SDE.  When the auto-completer is ready to attempt a completion, it calls the "typeRequest" function with the text the user has typed so far.  This is where we add the code to look up potential completions from the online SDE:

/* Support autocomplete of an EVE type */
var typeRequester = function(request, response) {
    // Query term uses SQL wildcards
    var queryTerm = '%' + request.term + '%';
    // Limit to 100 results for speed
    swagger.Inventory.getTypes({maxresults: 100, typeName: '{like: "' + queryTerm + '"}'})
.then(function(result) {
   var choices = [];
   for (var i = 0; i < result.obj.length; i++) {
}).catch(function(error) {
   console.error("Swagger promise rejected", error);

According to the jQuery docs, the auto-completer will pass a "request" object with a "term" property containing the text entered by the user.  Any potential choices should be returned by invoking the "response" object, which will be a callback taking a list of choices.

To lookup potential matches, we use the "like" query support provided by the online SDE.  You can read more about this feature (and other query choices) here.  Essentially, we're constructing a SQL wildcard match of the form '%term%' and invoking the "getTypes" endpoint to collect potential matches.  We use the "promise" calling style again and populate an array of choices based on the returned type names.  If we catch an error, we return an empty list (the jQuery library requires that we always return a result, even if an error occurs).  We also limit the number of responses to make the auto-completion appear a bit snappier.

Parting Words

That's all there is to it!  The same pattern is usable with other SDE calls as well.  For example, you can create a "region auto-completer" by replacing "swagger.Inventory.getTypes" with "swagger.Map.getRegions" (and other naming changes, like replacing "typeName" with "regionName").  Hope this helps.  Happy coding!

Thursday, June 9, 2016

EveKit Updated for New XML API Endpoints

The Citadel release (April, 2016) of EVE Online added two new API endpoints as documented here.  These two new endpoints, Clones and Skills, provide access to a subset of the normal character sheet for a character.  You can therefore restrict an EVE XML API key to only provide access to these new endpoints.  If your key already provides access to the full character sheet, then there is no point authorizing these new endpoints as they add no new information.

We've added support for these new endpoints in EveKit for those of you who want to provide more restricted keys.  Since the data provided is a subset of the normal character sheet, we decided to just partially populate the current character sheet structure instead of introducing completely new EveKit model objects.

Here's how the synchronization works:

  1. We first check whether your API key allows character sheet synchronization.  If it does, then we sync the character sheet as usual and we're done.
  2. If character sheet synchronization isn't allowed, then we attempt to sync both Clones and Skills - whatever your API key happens to allow.  If your key allows neither of those endpoints, then we're done.
  3. If your key allows Clones, then your character sheet will be updated only on the fields provided by this endpoint (all of your clone information is updated, along with some attributes like race, bloodline, etc).
  4. If your key allows Skills, then the "free skill points" field of your character sheet is updated, as is your skill list (which we store in EveKit as separate model objects).
We haven't added any new EveKit access key settings for these new endpoints.  If your EveKit access key allows character sheet access, then you'll be able to see all the information updated by the Clones and Skills endpoints.

Tuesday, May 17, 2016

Re-Introducing EveKit

EveKit was introduced in mid 2014 via this forum post.  Since that time, the features of EveKit have changed a bit and the service itself was shut down for several months as I reconsidered whether it was even worth running.  As I wrote about a month ago, I almost immediately regretted not having the full history of my EVE API data available somewhere online.  So I started on the path of bringing it back in a more sustainable way.  I may still be the only user, but my hope is that others will find the code useful, as well as the various applications I plan to build on top of EveKit.  EveKit is 100% open source.  You can find all the code on our GitHub site, along with basic instructions for running it yourself.

And so, without further ado, I'm happy to announce that EveKit is officially back in business.  The new site is located here:

The rest of this entry will describe the basic features of EveKit.  The site has pointers to more detailed documentation as well as "get started" guide right on the front page.  I'll also be updating the few videos I produced around the function of EveKit.

What is EveKit?

EveKit is a web application which downloads your EVE API data to a backend server and gives you access to your data through a REST interface, or via a bulk download (a zip of CSV files).  The hope is that EveKit will serve as a platform for other third party developers.  To that end, EveKit provides the following key features:

  • EveKit stores data for both characters and corporations.  You tell EveKit what kind of data to store when you add an EVE API key.
  • EveKit can pull data from the EVE servers on demand, or continuously (at least as often as the EVE API allows).  You can change this setting for an account at any time.
  • EveKit versions all data.  When a new version of some data item is downloaded, the old version is stored away with a pair of time stamps indicating the range over which it was "current".
  • EveKit data can be accessed through a standard REST API.  We also provide a Swagger specification to make it easy to use our API from various programming languages.
  • EveKit data can also be downloaded as a daily snapshot, which stores your data as a zip archive of CSV files.
  • EveKit lets you share your data with others using EveKit API keys.  These keys are similar to EVE API keys in that they can be used to limit accessibility.

What do all these features give you?
  • an always up to date and comprehensive copy of your API data.  No more worrying that you may miss data because your desktop application hasn't contacted the API in a while.
  • a complete history of your API data.  Want to know what your character sheet looked like several months ago?  No problem, just ask EveKit to return a view of your data at a given time.
  • a web accessible copy of your data, either for backup, or for use in other tools.  EveKit provides its own API keys for accessing the data it stores.  These keys can be provided to other applications for accessing your data.

How Do I Use EveKit?

The front page of the site has a "Getting Started" checklist, but the steps amount to this:

  1. Login using one of the following mechanisms (these all use OAuth, EveKit doesn't store user credentials):
    1. Your EVE Online account;
    2. A Google account; or
    3. A Twitter account.
  2. Create a new "Sync Account" (via the "Sync Accounts" menu).  You'll need a valid EVE API key for this step.
  3. Wait for your data to sync for the first time, either on-demand (if you chose manual sync), or within a few minutes (if you chose auto sync).

Using the APIs to access your data take a bit more work, but the gist is that you need to create at least one EveKit API key, then use the REST APIs with the credentials from your EveKit key to access your data.  Creating an EveKit API key is straightforward:

  1. Use the "Data Access Keys" drop down to select the account you want to create a key for (this menu is only visible if you are logged in).
  2. Click the "Create New Access Key" button.
  3. Follow the instructions on the new key page (e.g. name your key, drag over one or more sections you wish to allow access to).
  4. Keys are usable immediately once they have been saved.
Your model data is accessible using a REST API and the access key you just created.  Since we've annotated the API using Swagger, we can provide documentation using the Swagger UI.  You can view this documentation by selecting the "Model API" link (under the "API" menu).  That version of the documentation requires you to copy your API access key and credential in to each call.  If you want to browse the API with key and credential pre-populated with one of your keys, use the "Data Access Keys" menu to navigate to the list of keys for one of your accounts.  Then click the "Explore" button on the appropriate key.  This will bring up a version of the Swagger UI pre-populated with the appropriate key and credential.

Need More Help?

There are short videos on the EveKit Guides, Videos and Tutorials page that cover the account creation and data access instructions above.  You'll also find other useful information on that page, such as how to use Swagger to instantiate a model client in your favorite language.

If you have questions, or otherwise need more help, feel free to drop a comment here or post in the appropriate section of the Orbital Enterprises forum.

Tuesday, April 26, 2016

EVE Online Static Data Export Web Service

As part of supporting the EVE Online third party community, CCP regularly releases the Static Data Export (SDE) which is a collection of database and YAML files containing static data used in the game.  This data provides reference information used by various third party tools.

The raw form of the data export is a bit unwieldy to use because it uses a mix of data formats which are convenient for managing versions and building in-game data, but less convenient for typical third party use which involves SQL style queries.  As a result, several third party conversions have been created that transform the SDE into various SQL vendor formats.

The database formats are easier to use, but you'll still need to set up a database and import the data.  The evekit-sde project is an attempt to go one better and provide a web service which exposes a REST API for access to all SDE data.  The REST API is provided by Swagger annotated endpoints.  This makes it very easy to view high quality documentation for the REST API, try out API calls directly, or generate clients in a variety of languages.  We'll demonstrate each of these features in this blog post.


EveKit SDE Public Site
GitHub Project Page

Viewing SDE Documentation and Trying out the API

Following the link to the public site (see above) will bring you to a landing page which lists all the current SDE conversions that are exposed by the site:

Clicking on an SDE release will bring you to the landing page for that release:

and, finally, clicking on "Swagger UI demo" will pull up the Swagger UI:

The Swagger UI demo page is hosted by the Swagger project, but can be used to view any Swagger configuration with a URL.  In this case, we insert the URL for the SDE service.

The main Swagger view shows all of the SDE sections exposed as web service endpoints.  Clicking on a section will show all the endpoints for that section.  There is one endpoint for each table in a given SDE section:

Selecting an endpoint will show a description of the section and includes a form for making a call to the endpoint:

Every endpoint call contains two (optional) parameters:

  • contid - this is a simple offset into the endpoint call result list
  • maxresults - the maximum number of results to return from the call.  The max is 1000.
These parameters are optional with sensible defaults.  All remaining parameters are called "selectors" and are used to filter the results returned from the call.  A selector is a JSON string with the following syntax:

  • {any: <boolean>} - Wildcard selector. If true, then this data field is not used to filter returned data. Setting this value to false has no effect.
  • {like: <string>} - String match selector. If the associated data field is string valued, then all returned data must satisfy the SQL expression 'field LIKE selector'. Normal SQL 'LIKE' syntax is allowed (e.g. % as wildcard).
  • {values: [<v1>,...,<vn>]} - Set selector. The associated data field of each returned data item must contain one of the listed values.
  • {start: <lower>, end: <upper>} - Range selector. The associated data field of each returned data item must satisfy lower <= value <= upper.

This syntax allows very flexible selection on SDE data. For example, to quickly find the inventory category for ships, set the categoryName selector to {like: "%Ship%" } on the inv/category endpoint:

which results in the response (click "Try it out!" to generate):

To find all types with "Raven" in the name, set typeName to {like: "%Raven%"} on the inv/type endpoint; and, to further restrict this set to those types actually on the market, set marketGroupID to {start: 0, end: 1000000}:

When you click "Try it out!" you'll see the set of Raven associated types on the market.

There are a few details you should be aware of when making calls:
  1. There is no explicit indication that there may be multiple pages of results.  To page through results, you'll need to call an end point until the number of returned results is less than "maxresults".  You'll also need to adjust "contid" for each call.  Typical code will look like this:

    contid = 0
    maxresults = 1000
    results = <call endpoint>(contid, maxresults, selectors...)
    while (results.length < maxresults) {
      ...Do something with data...
      contid += results.length
      results = <call endpoint>(contid, maxresults, selectors...)

  2. You can cut/paste the "Curl" line from the Swagger UI to call an end point from the command line.
  3. Every call has three possible results:
    • 200 (OK) will be returned when the call to the end point succeeds.
    • 400 (Bad Request) will be returned if a selector has an incorrect format.
    • 500 (Internal Server Error) will be returned if the end point service encounters an internal error.
  4. HTTP response headers are used to relay SDE service meta-data:
    • Date - server time when the result was returned (UTC)
    • Expires - the date when the returned data will expire. Defaults to 24 hours for SDE results (since these are unchanged between releases)
    • EveKit-SDE-Version - the name of the SDE release for which results were returned
  5. Calling the API from the Swagger UI doesn't show the Date and EveKit-SDE-Version response headers (but they are there).  If you use the curl command line and pass the -v option you can view the HTTP response headers directly.

Creating an SDE Client

The Swagger Editor makes it very easy to generate a client in your favorite language.  You can find instructions for this process in a previous post (see the examples section toward the end).  You'll need the URL for the Swagger description of the SDE version you want to build a client for.  This can be found on the landing page of the SDE version (see above).

Parting Words

The SDE service we've described here will likely work best for third party applications that only need select pieces of the SDE.  The query capabilities of selectors makes it easy to select just the data you need.  Larger, more complicated tools, will likely be better served by biting the bullet and importing the SDE dump as a database.

Tuesday, March 1, 2016

EveKit Coming Back Soon

I shut down EveKit several months ago.  A non-trivial amount of data was being stored by EveKit (about 150 EVE accounts were storing data), but no one was really using the data.  I was using my own data, but setting up an entire site for this was overkill.  Also, it wasn't free.  There was enough data and resources being used on Google's cloud that this was starting to cost real money.

Since then, I've started a few new projects and realized I still need the main feature EveKit provided: an online store of my EVE character and corporation data, preferably with indefinite history.  For example, I recently wrote about a short fun project where I added some simple EVE-related features to an Amazon Alexa.  An obvious next step is to teach Alexa to tell me about my industry jobs; tell me how my market orders are doing; read me my mail, etc.  EveKit had all this data, and now I'm missing it.

So I've decided to bring EveKit back in a simpler and easier to maintain form.  The key features will still be there, namely:
  • A service to automatically download and store API data for characters and corporation;
  • Full history for all data: you can view your data as it appeared at a particular point in time;
  • Various APIs for accessing your data; and,
  • A batch download option to get all your data in one dump.
The original EveKit also exposed the SDE as a service.  I'll be doing that as well, but as a separate service not directly associated with EveKit.  There's a market out there for the SDE independent of EveKit.  Might as well try to fill that need too.

Lastly, all the code will be open source.  I've already started releasing some of the key pieces on the Orbital GitHub page.  Look for the projects starting with "evekit".

I'll be posting further updates as the release gets closer.

Saturday, January 16, 2016

Holiday Hack Project: Using Alexa to Access EVE Online Data

Most people have heard about Amazon's Echo in-home digital assistant.  Echo comes installed with a digital assistant named "Alexa", and will do the things most digital assistants do, like answer random questions, set alarms, keep to-do lists, etc.  However, Amazon went one step further and released public APIs for adding your own capabilities to Alexa.  Amazon has two offerings here:
  • The Alexa Skills Kit (ASK) is an API for adding "skills" to Alexa.  Skills are just named features that you can ask Alexa to use.  Custom skills are hooked up to web services you write to implement the skill.
  • The Alexa Voice Service (AVS) is an API for embedding Alexa in custom hardware.  Your device needs a microphone to accept voice input which you forward to the Alexa Voice Service.  The voice service passes your voice input to Alexa, then sends back an audio response which you can play through your device.
These two offerings are in developer preview at the moment, which means they're free.  So as I had a bit of time this holiday to do some hacking, I decided to see if I could add an Alexa skill which handled queries about EVE Online.  Here's an example of what I was able to do:

It turns out creating an Alexa skill is pretty easy, but Alexa has a hard time understanding things which aren't in the dictionary (e.g. a good portion of the in-game objects in EVE) so there's still more work to do to make this truly awesome.  In this post, I'll walk through all the steps to create an Alexa skill for serving up EVE data.


Alexa Skills Kit Basics - Skills, Intents and Architecture

Our goal here is to train Alexa how to respond to a few simple requests related to EVE Online.  To do that, we need two things: a "skill" and a skill handler.  A skill is a definition of the types of spoken requests we'll respond to.  A skill handler is code which will receive matching requests and generate appropriate replies.

An Alexa skill has four basic components:
  1. A name.  This is the trigger word you'll use to invoke your skill through Alexa.  Pick something easy to pronounce (and therefore, easy for Alexa to recognize).
  2. A schema with one or more intents.  An intent represents an action that matches a user's spoken request.  An intent can have placeholders, called slots, which match a set of words and act like arguments to the intent.  Alexa has several built in slots, but you can also define your own.
  3. Zero or more custom slots.  A custom slot is used when you need an argument that can match a specific set of words not covered by a built-in slot.  You typically use a custom slot to define a domain specific argument, for example the set of all in-game items in EVE.
  4. Finally, every intent must have one or more sample utterances.  A sample utterance gives an example of a user request which matches an intent, including the position of any slots.
Of the components of a skill, custom slots and sample utterances are perhaps the most important as they are used to train Alexa how to recognize an intent provided by your skill.  The more sample utterances you provide (covering a variety of ways to match an intent), the more likely it is that Alexa will be able to match a request to an intent.  We'll show a few examples of defining skills in the next section.

Once we have our skill defined, we'll need to implement a skill handler.  A skill handler is either a web service or an Amazon Web Services (AWS) Lambda function which accepts a structured skill request and returns a structured reply.  For this writeup, we're using a web service.  We'll show some sample code below.

Skill handling in Alexa uses the following flow:
  1. A user makes a spoken request invoking the name of a skill, for example: "Alexa, ask Auren for the current time".  In this example, "Auren" is the name of our skill (see below) and "(ask) the current time" is the user's request.
  2. Alexa recognizes our skill name and analyzes the request.  If all goes well, Alexa will match the request to an intent defined for our skill, filling in any defined slots with the appropriate words from the request.
  3. Alexa packages the request into a structured format (JSON) and forwards the request to our skill handler.  The skill handler will process the request and generate either a speech reply, or a "card" reply (or both).  These replies are returned to Alexa for processing.
  4. If our skill handler generated a speech reply, then Alexa will stream speech audio to the device where the request was made.
  5. If our skill handler generated a "card" reply, then Alexa will render the card in the companion application for the device where the request was made.  If the device is an Echo, then your phone is usually the companion app and you'll see your card rendered there.  You can also see your card on the Alexa web front end.
There are three basic error cases that are handled as follows:
  1. If Alexa can't understand your skill name or match a request to an intent, it will tell you so through your device without ever sending a request to your skill handler.
  2. If Alexa can't reach your skill handler, it will tell you so through your device, usually with a message along the lines of "that function is not available right now".
  3. Finally, Alexa may match your intent but not the way you expected.  In that case, your skill handler may report an error message which Alexa will report back through the requesting device.
So much for theory.  In the next section, we'll build our skill and our first intent.  By the way, all the code and setup instructions are in our GitHub project here.

Introducing Auren - Aura's Less Helpful Cousin

Skills need a name that Alexa can recognize to trigger your intents.  Keeping with the EVE theme for this project, I chose "Auren" (pronounce like Oren) because it sounded kind of like Aura, the in-game assistant in EVE.  However, Auren isn't (yet) as helpful as Aura.

For this project, we'll teach Auren to do four things:
  1. Provide help, when requested, to explain what Auren can do.  All skills should do this.
  2. Report the current time in EVE.  This is a simple starter skill to demonstrate the process.
  3. Report EVE server status.  A slightly more complicated skill that adds in calling the EVE XML API servers.
  4. Report the current price in Jita for an in-game item.  This skill will call eve-central's API to look up and report price data.  What makes this skill more complicated is that we'll need to use voice recognition to figure out which in-game item should be retrieved. 
We'll cover three steps in building our Alexa skill:
  1. Defining intents and sample utterances.
  2. Implementing a web service to handle intents.
  3. Testing the skill.

 Defining Intents and Sample Utterances

We have our skill name "Auren", now let's create an intent schema.  This is just a JSON document naming the types of requests your skill will handle.  Here's what our schema looks like for handling the first three items on Auren's list:
  "intents": [
      "intent": "AMAZON.HelpIntent"
      "intent": "ServerTimeIntent"
      "intent": "ServerStatusIntent"
The first intent is an Amazon built in intent that matches requests for help (e.g. "Alexa, ask Auren for help").  We don't need to provide sample utterances or anything else for a built-in intent.  We just have to be ready to process this request if it shows up at our web service.  A list of built-in intents is here.  The second and third intents name requests for EVE server time or EVE server status, respectively.

Our first three intents don't have any slots (custom or otherwise), so we can move on to sample utterances.  Since utterances for built-in intents are handled automatically, we only have to provide utterances for the "ServerTimeIntent" and the "ServerStatusIntent".  These are defined as lines in a simple text file.  Here are the utterances we created for this project:
ServerTimeIntent time
ServerTimeIntent current time
ServerTimeIntent eve time
ServerTimeIntent eve online time
ServerTimeIntent what is time
ServerTimeIntent what is current time
ServerTimeIntent what is eve time
ServerTimeIntent what is eve online time
ServerTimeIntent what is the time
ServerTimeIntent what is the current time
ServerTimeIntent what is the time in eve
ServerTimeIntent what is the time in eve online
ServerTimeIntent what is the current time in eve
ServerTimeIntent what is the current time in eve online
ServerStatusIntent server status
ServerStatusIntent eve online status
ServerStatusIntent eve server status
ServerStatusIntent what is eve's status
ServerStatusIntent what is the current server status
ServerStatusIntent current status
ServerStatusIntent current server status
ServerStatusIntent tell me current status
ServerStatusIntent tell me current server status
ServerStatusIntent tell me eve's status
A sample utterance consists of the name of the intent following by a sentence snippet giving a sample request.  The more samples you provide, the better Alexa will train to match your intent.  There are a few basic rules you need to follow when specifying an intent, but there are no stated limits on the number of sample utterances you can provide.  We've provided a decent set above, but we're missing a few you might hear (e.g. "can you tell me the time?").  Thus, there's a small chance Alexa won't match our intent correctly.  Alexa saves all incorrect translations so you can always go back later and improve the utterance list.

Implementing a Web Service to Handle Intents

Now let's take a look at our web service which will handle requests.  Note that you can instead use an AWS Lambda function to implement your skill (this eliminates some complications with certificates), but I elected to use a Java servlet because I already have a server available and Amazon provides a servlet-based library as part of the Alexa Skills Kit.

If you go the web service route, there are a few requirements.  The important requirements for testing are:
  • Your service must be accessible via the Internet; and,
  • Your service must accept requests on port 443 (https) and use an Amazon trusted certificate.
You'll need to worry about the rest of the requirements in order to deploy your skill beyond development mode.  We're not doing that here, so we'll skip the other requirements.

The Alexa Skills Kit Java library provides a ready made servlet ( which routes incoming requests to one or more "Speechlets".  A Speechlet is an interface ( which defines four operations:
  • onSessionStarted - called when a new session starts as a result of a user invoking your skill.
  • onSessionEnded - called when a session ends either due to a user ending the session, or not interacting with your skill.
  • onLaunch - entry point when a skill is invoked without an intent (e.g. "Alexa, start Auren").
  • onIntent - entry point for handling speech related requests.
The only operation we're interested in right now is "onIntent".  The other operations are life-cycle events which would be important in a more complicated skill, but aren't relevant here.

Let's take a closer look at the handling of the ServerTimeIntent.  This intent will arrive on the onIntent call, so we'll need to implement that method first.  We can start as follows:
public SpeechletResponse onIntent(final IntentRequest request, final Session session) throws SpeechletException {
    Intent intent = request.getIntent();
    String intentName = (intent != null) ? intent.getName() : null;
    if ("ServerTimeIntent".equals(intentName)) {
      return getServerTimeResponse();
    } else {
      throw new SpeechletException("Invalid Intent");
It's good practice to verify the intent name is not null so we do that here.  If the intent matches ServerTimeIntent then we return an appropriate response.  Otherwise, we throw an exception.  Here's what the response generator looks like:
private SpeechletResponse getServerTimeResponse() {
    // EVE time is always UTC so just return that directly
    Date now = new Date();
    SimpleDateFormat formatter = new SimpleDateFormat("HH:mm");
    String speechText = "The current EVE time is " + formatter.format(now) + ".";
    // Create the Simple card content.
    SimpleCard card = new SimpleCard();
    card.setTitle("EVE Online Time");
    // Create the plain text output.
    PlainTextOutputSpeech speech = new PlainTextOutputSpeech();
    SpeechletResponse result = SpeechletResponse.newTellResponse(speech, card);
    return result;
A standard response will provide both a card as well as text to be spoken by Alexa.  We use the same text in both cases, namely just the current time UTC which also happens to be EVE time.  The call to "setShouldEndSession" indicates that this response ends the current session.  You would set this to false if your response required further action by the user (e.g. your response asked the user for more information).

Once we've written our Speechlet, the last step is to add it to the Amazon provided servlet so that incoming requests are handled properly.  This is done by sub-classing and adding our Speechlet in the constructor:
public OrbitalAurenServlet() {
    // Add our speechlet
    this.setSpeechlet(new OrbitalSpeechlet());
You can see the final version of our servlet and Speechlet here.

Testing Auren

We've defined intents and sample utterances, we've implemented a Speechlet to handle intents, now we're ready to deploy and test our skill.  We'll skip the step of deploying our servlet to a servlet container.  There are instructions on how to do that on our GitHub page.

To test our new skill we need to create a configuration on Amazon's development site.  Once you've created an Amazon developer account and logged into the developer site, you can navigate to the Alexa Skills Kit page to start configuring a new skill.  Clicking on "Add a New Skill" brings up a page like the following:

Adding a new Alexa skill
Completing the configuration of an Alexa skill involves a six page form but only the first three pages need to be configured for testing.  The important fields on this first page are the name (which will appear in the Alexia companion app), the invocation name (the word users will use to tell Alexa to send your skill a request), and the endpoint.  As you can see from the figure, we've filled in "auren" as our invocation name and configured an HTTPS endpoint.  When you save this page for the first time, an "Application Id" will be created for you which you'll need to provide to your servlet (see GitHub page for details).

The next page is where we specify the interaction model:

Configuring a skill interaction model
The view in the figure represents our final interaction model.  Configuration is just a matter of cut and paste from the appropriate files containing your intent schema, values for any custom slots, and your sample utterances.  We'll talk about custom slots in the next section.

The next page you need to configure is actually the SSL Certificate page but we'll skip that one for brevity.  Suffice it to say, you can get away with a self-signed certificate for testing, but you'll need a certificate signed by a valid certificate authority in order to deploy beyond development mode.  We happen to have a valid SSL certificate for the endpoint we specified on the first page, so we're good to go.

The fourth page is where we're finally able to test our skill:

Skill test page
This page lets you enter a test string and verify that your skill is responding properly.  Here's what it looks like for getting the current time:

Ask Auren for EVE time!
On the left is the request Alexa sent to our servlet.  On the right is what our servlet sent back.  Notice we sent back both a card and a speech response.  If you click the "Listen" arrow you'll hear Alexa speak your response.

Of course, the whole point of this was to make the skill available on an Alexa device like Echo.  Fortunately, Amazon provides a really nice way to do that: if your Echo device is registered under the same login as your developer account, then all your skills in development are automatically added to your Echo!  This means we can walk up to our Echo and ask it a question.  Here's the video again of me doing exactly that to get EVE server status:

When you invoke your skill directly on Echo, you'll also generate a card on the Alexa site.  This is a great way to debug if Alexa has trouble translating your speech to text.  Here's what our card looks like for the request in the video:

The card corresponding to our voice request
The top part of the card shows the card response we sent back.  The bottom part shows how Alexa parsed the voice request.

The first three intents are implemented in the same way (help, server time and server status).  Getting the current price of an item in Jita is a little more complicated (and also doesn't work as well).  We'll cover that next.

Auren's Next Trick - Getting EVE Market Prices

The last intent we'll implement is a request to Auren to get the market price of an EVE item from Jita.  We need to do two things to get this working:
  1. We need to update our interaction model so that we can recognize requests for EVE items; and,
  2. We need to add code to our servlet to request market data prices.  
Let's look at the interaction model first.  For the market data intent, we need to capture the name of the item the user wants to price which means we need a slot.  Moreover, many EVE items aren't in any dictionary, so we'll need a custom slot to help Alexa understand the user's request.  Here's what the intent looks like:
  "intents": [
      "intent": "MarketPriceIntent",
      "slots": [
          "name": "Item",
          "type": "EVE_MARKET_ITEM"
We name our intent "MarketPriceIntent" and add one slot called "Item".  This is a custom slot, so we'll give it the name "EVE_MARKET_ITEM".  If this wasn't a custom slot, you'd use one of the built in slot types described here.

We'll need to tell Alexa more about our custom slot, but before we do that let's look at the sample utterances so we can see where we'll use the slot:
MarketPriceIntent what is the price of {Item}
MarketPriceIntent price for {Item}
MarketPriceIntent price {Item}
MarketPriceIntent how much does {Item} cost
MarketPriceIntent what is the current price for {Item}
MarketPriceIntent current price for {Item}
MarketPriceIntent the price for {Item}
MarketPriceIntent the price of {Item}
MarketPriceIntent price of {Item}
MarketPriceIntent current price {Item}
MarketPriceIntent what is the price of a {Item}
MarketPriceIntent price for a {Item}
MarketPriceIntent how much does a {Item} cost
MarketPriceIntent what is the current price for a {Item}
MarketPriceIntent current price for a {Item}
MarketPriceIntent the price for a {Item}
MarketPriceIntent the price of a {Item}
MarketPriceIntent price of a {Item}
As you can see, all we need to do is add "{Item}" to the location where we expect the user to name the item they're interested in.  Simple, right?

Now back to our custom slot.  To create our custom slot we'll need to go back to the service configuration form on the "Interaction Model" page.  Clicking the "Add Slot Type" button brings up the following screen:

Creating a custom slot
"Type" is just the name of our custom slot, i.e. "EVE_MARKET_ITEM".  The values section is where we enter the text we want to recognize as members of our custom slot.  This is where we hit our first difficulty.  Currently, custom slots are limited to 5000 unique values across all the custom slots we define for a skill.  However, there are around 11000 unique EVE items in the market.  To work around this, I did a slightly fancy selection of 4000 EVE items from the Static Data Export.  My selection criteria was based on market group ID, where I picked the market groups players are usually interested in (e.g. ships, ammo, materials, etc).  You can see my final list here.  Not to worry though.  Even if Alexa can't match what the user says to a value in your list, it will often continue to send whatever it heard to your skill handler.  So in many cases we'll still be able to try to match what the user said to a value.  This suggests another way to get around the 5000 value limit which I'll discuss further below.  Clicking "OK" saves the slot definition.  This takes a few minutes for large custom slots as Alexa prepares the data for voice recognition.

Adding an intent, custom slot, and new sample utterances finishes the configuration of the market data request feature.  Now we need to implement the feature on our servlet.  We follow the same approach as before where we add an appropriate response generator to our "Speechlet", but this time we need to match a slot value to an EVE market item.  Let's look at the "onIntent" entry point first (full code is here):
  public SpeechletResponse onIntent(final IntentRequest request, final Session session) throws SpeechletException {
    Intent intent = request.getIntent();
    String intentName = (intent != null) ? intent.getName() : null;
    if ("MarketPriceIntent".equals(intentName)) {
      return getMarketPriceResponse(intent.getSlot("Item").getValue());
    } else {
      throw new SpeechletException("Invalid Intent");
The "getSlot" method pulls out the value Alexa heard when our user invoked this intent.  The "getValue" method converts the value to a string.  We then pass this value to our response generator which is slightly more complicated this time around:
  private SpeechletResponse getMarketPriceResponse(String item) {
    // Clean up the item
    item = item.toLowerCase();
    // This is what we return if we can't figure out what the user wanted
    String speechText = "I'm sorry, I couldn't find an EVE market item matching that name.  Either that item isn't on the market, isn't in my database, or I don't understand your pronunciation.";
    // typeNameMap is a Map<ItemName, ItemTypeID>
    if (typeNameMap.containsKey(item)) {
      // Found it, look for a price
      int typeID = typeNameMap.get(item);
      MarketStatApi api = new MarketStatApi();
      try {
        List<MarketInfo> info = api.requestMarketstat(Collections.<Integer> singletonList(new Integer(typeID)), null, null, null, 30000142L);
        if (!info.isEmpty()) {
          MarketInfo data = info.get(0);
          Double bid = data.getBuy() != null ? data.getBuy().getMax() : null;
          Double ask = data.getSell() != null ? data.getSell().getMin() : null;
          ... clean up these values a bit ...
          speechText = "Jita is reporting ";
          speechText += bid == null ? "no current bid" : "a bid of " + bidPrice.toPlainString() + " isk";
          speechText += " and ";
          speechText += ask == null ? "no current ask" : "an ask of " + askPrice.toPlainString() + " isk";
          speechText += " for " + item + ".";
        } else {
          // eve-central didn't have a price for this item
          speechText = "I'm sorry, there is no market information available for " + item + ", perhaps this is a rare item?";
"Empty market info list for \"" + item + "\"");
      } catch (ApiException e) {
        ... handle error calling eve-central ...
    ... Create card and text response as before ...
To look up market data, we need to map the user requested item to a type ID (the eve-central API looks up price data by type ID).  To keep things simple, we're using a simple table lookup based on the string value of what the user requested.  We populate our map from a larger type ID text file we extracted from the Static Data Export (full file is here).  If we manage to map the user's request to a type ID, then we call eve-central to look up the price.  The "MarketStatApi" class is a client we generated from a Swagger description of the eve-central API.  You can find more details about that on our project page.

 So does it work?  Let's try it from the Alexa development test page.  Here's an example where we get the current price of a Raven:

Getting the current price of a Raven
Hey it works!  Now let's try a real test on Echo:

So far so good, if a little on the verbose side.  Now let's see what happens when we ask the price of something that we left out of our custom slot.  We left Veldspar out, so let's try that.  First with the text interface:

Getting the current price of Veldspar
Good, the text interface doesn't care that "veldspar" wasn't included in the custom slot list.  Let's see if it works through the Echo:  Let's take a look at the Alexa card to see what it thought we said:

How Alexa translated "Veldspar"
So "Veldspar" became "builds bar".  Interestingly enough, Alexa may even fail before it gets to Auren:

In this case, the card shows it missing both "Auren" and "Veldspar":

Alexa misses both "Auren" and "Veldspar"
So it appears that if you can't get your word into the definition for a custom slot, then it's hit or miss as to whether you'll get something usable as an intent argument.

I said earlier that knowing that Alexa is good at translating dictionary words could be a way to fit within the word count limit of a custom slot.  The trick here is to remove all the words from your custom slot definition that are already in the dictionary.  This works because Alexa will fill a custom slot with whatever is translated, even if the word wasn't in your custom list.  So that's one trick we could try to give a better chance of capturing all the in-game items.  Another approach is to incorporate Alexa's strange translations of some words.  For example, we could make "builds bar" a synonym for "Veldspar" in our item map.  Those are improvements that would have to be made over time as we gather more data on how Alexa translates non-dictionary words it doesn't understand.

Last and perhaps one of the hardest problems to solve is accounting for the variety of ways people pronounce EVE in-game items.  This problem is enhanced by the fact that EVE has a large international community with unique pronunciation according to country of origin, as is evidenced by just about every episode of How Do You Say It:

So there's a lot more work to do to make Auren truly awesome.  We'll keep experimenting and post our results in a future blog.


If you can write a simple web service, then creating an Alexa skill is very easy.  A great way to start is to clone the orbital-auren-frontend project on GitHub and customize it to your liking.  Amazon did a great job making it easy to test your skill.  The fact that your skill is automatically available on your Echo, even in development mode, is a great feature.

Where we ran into trouble with Alexa is with large sets of non-dictionary words that we couldn't add to a custom slot definition.  Maybe such large sets are rare, but hopefully in the future Alexa will lift the restriction on the set size for custom slots.

We have a few ideas as to how we might make Auren better at handling EVE data, but we'd love to hear from others as well.  What cool features would you like to see in a voice interface?  We'll keep plugging away and post our results in a future blog update.