Monday, July 4, 2016

A site refresh including new Market Data endpoints

We released a refresh of the main EveKit site today.  The quick summary is:

  • The (now) four main offerings of the site are now accessible from the front page.  They are:
    • Account Synchronization - the original purpose of EveKit.
    • XML API Proxy -  a simple proxy to the EVE Online XML API endpoints.
    • SDE - Info and links to the online SDE versions that we host at EveKit.
    • MarketData - NEW! Info and links to the online EVE MarketData we host.
  • There's a nice little navigation bar in the top left corner that makes it easy to switch between offerings.
  • The new exciting feature is market data.  Some brief notes about that below.


EVE Online Market Data Access

About a year ago CCP exposed CREST endpoints for retrieving market data for EVE Online.  Recently, they've made many improvements to these endpoints, most notably providing the ability to download complete market data for a single region in one bulk download (technically, this bulk download would consist of several pages for a non-trivial region, but you get the idea).

Here at Orbital Enterprises, where our mantra is essentially "keep history for everything", we decided to start capturing this data for possible future use in EVE market play (region arbitrage, market cornering, station trading, etc).  To do that, we created a market data scraper which runs 24/7 with a daily archive run to package and store scraped data into a more convenient format.  We've built our scraper on top of our existing EveKit stack, the result of which is the EveKit MarketData Project which you can download from GitHub and try for yourself.

Scraping market data is good but one problem is that archive data and live data are in two different places (and two different formats): archive data is stored in Google Cloud Storage (see the EveKit MarketData Overview for more info) in a format convenient for certain types of extraction; but the most recent data is on the server in a snapshot format waiting to be archived at the end of the day.  This is kind of a pain if you're trying to develop an online tool which uses market data.  So we built a simple market data server which provides per-type, per-region access to either order books or market history on any given date we support (for book data, back to 2016-05-31; for market history, back to 2015-04-01).  The result of that effort is the EveKit MarketData Server Project which you can try out live here.

As a quick demonstration, we built a simple market viewer using our market data server.  You can try out the viewer here.  The code for the viewer is provided in the EveKit Frontend.

Using Market Data Offline

We'll be publishing an article or two on how to use EVE market data archives for developing trading strategies in the near future.  Those articles will be based on using Python and Jupyter notebooks.  In the meantime, the best documentation for how to retrieve and use EveKit market data archives offline is on the EveKit site itself here.


Post a Comment