LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dialpemo

Hi,

I have been trying to use the Http node to retrieve data from yahoo finance, but so far I only get the html code of the page it self, Is there a way to specify the specific data that I want to extract instead of extracting everything.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: simon.wilby

Hi dialpemo,

Attached is an example graph that queries yahoo finance. I used the following page as a guide for the GET request. http://developer.yahoo.com/finance/company.html
On that page is a link to the Symbol lookup that can be used to find more options on their supported http requests.

Hope this helps

Regards

Simon
Attachments:
yahho_finance_example.brg

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dialpemo

Dear Simon,

Thank you for your promptly response, that is quite good, is there a more generic method, if there is not an rss feed, for example I want to extract the data that is in in the table of the following link,

http://uk.finance.yahoo.com/q/hp?s=%...=01&f=2014&g=d

I keep getting an error saying that ">" is missing in one of the lines.

Please help

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: simon.wilby

Hi dialpemo,

I've had a quick look round, the best solution is to use yql (yahoo query language) as described in the following link http://www.jarloo.com/get-yahoo-fina...-data-via-yql/
from a quick search there seems to be a number of different ways to access the yahoo finance data.
http://stackoverflow.com/questions/1...istorical-data
http://stackoverflow.com/questions/3...google-finance

Here's the url sampled from the jarloo link:
http://query.yahooapis.com/v1/public...tableswithkeys

hope this helps

Simon

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: simon.wilby

Hi dialpemo,

I've just found this that should meet your needs:

here's how you can get started exploring the Yahoo! Finance tables in YQL:

Start at the YQL Console
Under the Data Tables section on the right, click Show Community Tables
Type finance in the search field
You'll see all the Yahoo Finance tables (about 15)

http://developer.yahoo.com/yql/conso...22010-03-10%22

Regards

Simon

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dialpemo

Simon thank you, that pointed me in the right direction.
I have a more generic question is there a way within Lavastorm to extract specific data from a website without the use of an api?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: simon.wilby

glad I could help.
I don't believe there is a way to extract specific data from a website at this time.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dialpemo

ok Cool thank you Simon, I really appreciate all your time and help

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ejones

Originally posted by: dialpemo
					

I have a more generic question is there a way within Lavastorm to extract specific data from a website without the use of an api?
			

I'd like to understand the question better. There are many nodes other than the HTTP node that can be used to extract data. I'm thinking of the database query nodes, Java and Python nodes. There are also nodes that can read in CSV and other file types so if the data were already in a file it could be read. Is there any type of way to method for extracting data you had in mind?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hi,

I guess the response to your question is that if you are using the HTTP node and pointing it to a webpage you are effectively requesting just the HTML data that the specified page returns.
If you use a web services API, then this will likely return XML or JSON data which is what you are looking for.
You can then obviously use the XML Data or JSON Data nodes to process the data.

If you can only retrieve the HTML, it is still possible to parse out the contents of the HTML.
There are probably some 3rd party APIs that you can find out which do this sort of screen-scraping.
However, since HTML is designed to describe the presentation and not the data layer, this is likely time consuming and error prone.
Every time the layout of the page you request changes, you will need to change the way you process the data.

This is the reason why web services are used - to extract the data & perform related operations and not worry about the presentation layer and how this information should be displayed in a browser.

If not such web services exist, then you are left with the HTML and there isn't much that can be done about this outside of putting in requests to whoever runs the site you are accessing to provide such an API.

Regards,
Tim.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dialpemo

HI Tim,

Quick question I have tried different methods to use the Http node to login to Dropbox or even my email, but I haven't been able to pass the login stage. The reason I am asking is because I have been approached to get some data from the Northern Thrust website, from the asset side for a client, but I haven't got their login details as they are unique to this person so I am trying to check with other sources to later on replicate the procedure.