Downloading Market Data

Forum rules
This forum can be viewed by the public.
<<

Xenesh Simmons

Portrait

Member
Member

Post 2017.10.14 05:46

Downloading Market Data

Hi all, hope your day is going well.

I'm looking into getting into the market game. I have a math/stats background so I am particularly interested in doing a massive amount of number crunching and market analysis. Dunno if it'll work but just the challenge interests me. Unfortunately, I need the numbers, does anyone know where I can get them?

I can parse XML data from places such as eve-marketdata using their API (e.g.https://api.eve-marketdata.com/api/item_prices2.xml?char_name=demo&region_ids=10000002&buysell=s) but as far as I can tell, this only provides the latest updated numbers. I am particularly interested in past data such as what is graphically shown herehttps://eve-marketdata.com/price_check.php?type_id=34&region_id=10000002&solarsystem_id=&type=history&format=graph but with the granularity of a per order basis. I have a feeling it may not exist/publicly accessible but figure I would ask around.

If anyone has any insight, I'd love to hear it. If I make any headway I may post in the future, depends how lucrative it is ;)

X
<<

Turhan Bey

Portrait

Director of Logistics
Director of Logistics

Post 2017.10.14 20:09

Re: Downloading Market Data

If it exists, then sites like eve-marketdata and eve-central are the most likely to have it. I'd imagine it would be in some archival download form due to the data size, though, instead of an API call.

But, I have not looked for historical data of that detail myself, so I can't say if it exists or not. In case it doesn't, you may wish to begin collecting it yourself using current data. Or perhaps some other enterprising statistician has already been doing this and may speak up here soon. :)
----

EDIT: There's also the eve-central daily dump archive, which seems like it may be what you're looking for but in an unprocessed state, but that's been down for quite awhile now. They were hoping to come back by the end of last month, but it's already halfway through October.
ImageImageImage
"The first mistake most gameplay zealots make is thinking that the game should revolve around them. The second is believing that it already does."
"Maybe it's just a game. Or perhaps it's a world. Perhaps it's a community. One thing is for certain: If it's only a game to you, then that is all it will ever be."
<<

Ann'Dra Padecain

Portrait

Member
Member

Post 2017.10.15 07:17

Re: Downloading Market Data

o/,
Looks like following site has historical data or a means to extract: https://www.fuzzwork.co.uk/market/link/10000002/3165
They also seem to offer an API... Maybe worthwhile scratching in there?
Good luck
<<

Garrett Achasse

Portrait

Member
Member

Post 2017.10.17 16:07

Re: Downloading Market Data

Xenesh Simmons wrote:I am particularly interested in past data such as what is graphically shown herehttps://eve-marketdata.com/price_check.php?type_id=34&region_id=10000002&solarsystem_id=&type=history&format=graph but with the granularity of a per order basis. I have a feeling it may not exist/publicly accessible but figure I would ask around.


Are you asking if it's possible to see the data for each individual fulfilled buy and sell order? That level of detail is not available. The lowest level of granularity you can get is what is provided on the price history tab of the market and that is an aggregate the entire region. I've recently started playing with trading and I was also looking for the same details you were.

Have you checked out the program Evernus? It provides a lot of market data and tools and may be a place to start. I'm using it and in all honesty it is way over my head. I know there is a lot to it that I do not fully understand but it's helped me to stumble through trying out station trading and inter-region trading.
<<

Stasia Ohau

Portrait

Member
Member

Post 2017.10.18 14:39

Re: Downloading Market Data

Garrett Achasse wrote:The lowest level of granularity you can get is what is provided on the price history tab of the market and that is an aggregate the entire region. I've recently started playing with trading and I was also looking for the same details you were.

That's partially true.

Yes the history endpoints only provide aggregate data but the active order endpoints do expose individual orders. With active polling you can derive the sales. The one potential gotcha is if the complete order disappears did it sell out or was it cancelled? Another thing I haven't tested is if the order_id changes if the order is modified.
<<

Garrett Achasse

Portrait

Member
Member

Post 2017.10.19 13:49

Re: Downloading Market Data

I don't have the expertise to know for sure whether you can parse out the data of individual buys and sells, but it seems like if it was possible someone would have already done it.
<<

Stasia Ohau

Portrait

Member
Member

Post 2017.10.19 14:36

Re: Downloading Market Data

Of course it has. That's the sort of market intelligence that people keep for themselves. :smug:
<<

Xenesh Simmons

Portrait

Member
Member

Post 2017.10.20 05:50

Re: Downloading Market Data

Hey everyone, thanks for the info. I had a bit of real life incident so sorry for lack of response.

At the moment I've diverted focus slightly to just building a tool for finding opportunities in the production market as a semi-passive income stream (since this doesn't require the same granularity of market data) while I build my own API to handle a more rigorous Market Prediction program as well as hone in some of the math (developing null market models and the fractional calculus and partitioning tools to find mesoscopic networks so I can try and implement some sort of directed cross-correlation model). But I'm also looking into your suggestions to see if there's something I've missed.

Turhan Bey wrote:EDIT: There's also the eve-central daily dump archive, which seems like it may be what you're looking for but in an unprocessed state, but that's been down for quite awhile now. They were hoping to come back by the end of last month, but it's already halfway through October.


I'm very much interested in what Eve-Central has but yeah, unfortunately they're down :/

Turhan Bey wrote:But, I have not looked for historical data of that detail myself, so I can't say if it exists or not. In case it doesn't, you may wish to begin collecting it yourself using current data. Or perhaps some other enterprising statistician has already been doing this and may speak up here soon.


I've been scouring the internet for such an enterprising statistician. Unfortunately, since Eve has been around for so long there are just so many dead ends. Projects that became inactive in 2012, 2013, 2016, etc.

Ann'Dra Padecain wrote:o/,
Looks like following site has historical data or a means to extract: https://www.fuzzwork.co.uk/market/link/10000002/3165
They also seem to offer an API... Maybe worthwhile scratching in there?
Good luck


Thanks! This was helpful. I have moved on to using the Eve Swagger Interface https://esi.tech.ccp.is/latest/ just cause testing found it to be faster and I just trust CCP to have better uptimes/etc.

Garrett Achasse wrote:Have you checked out the program Evernus? It provides a lot of market data and tools and may be a place to start. I'm using it and in all honesty it is way over my head. I know there is a lot to it that I do not fully understand but it's helped me to stumble through trying out station trading and inter-region trading.


I have! It's an excellent program that I've been messing around with. I'm currently looking at finding out how they derive their data (such as OHLC) since it's not apparently accessible from current APIs (to my knowledge). The program itself doesn't suit me completely since I want to run my own predictive algorithms in R or Python and eventually have it running continuously on a little Raspberry Pi i'm setting up.
<<

Zhorack Jaynara

Portrait

Member
Member

Post 2017.12.29 17:15

Re: Downloading Market Data

Hi Xenesh,

Have you been having any luck with Eve Swagger Interface?

I'd love to work with someone on building a web interface to use ESI (including character stats) to assist in:
1. Station Trading
2. Market Analysis for mining and PI
3: Manufacturing (T1 and T2), including mapping T2 and T1 blueprints to their required inputs and costs.
With those mappings, show how station-traded items relate to mined and PI (as current character) items to find best value versus time.

Would this tie into your work at all?

Thanks,

Zhorack

Return to Research, Production and Trade

Who is online

Users browsing this forum: No registered users

Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Powered by Dediserve