LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: jpstory

Hi All

My daily process involves getting data (a table) from centain website.

Can someone show me how to set it up? For example, Can you please set up the ndoe so I can get the table from the following link:
https://strikeprice.cftc.gov/StrikePriceFormat.aspx

Thanks

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: awilliams1024

Hi jpstory,

Wherever possible it is best to obtain data such as this in a standardized format such as XML or JSON data rather than web scraping the data from a HTML page as the latter approach is inherently brittle and will break if the format of the data on the page changes. However, with that caveat, here is a data flow that uses the HTTP node to get the raw page response data and then uses a Filter node and scripting to extract the data from the embedded table.

HTTP_Node_Example_StrikePrice_Data.brg

Regards,
Adrian

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: jpstory

Hi aWiiliamns1024

Thanks for your help, appreciate it!

I got an error while trying to run your sample graph. The error says: "Error occurred during executing request: Unknown host: strikeprice.cftc.gov"

As you suggested, it's better to obtain data in XML or JSON, can you also provide a sample graph using such method, thanks!

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: awilliams1024

The HTTP node worked for me.

HTTP_Node_Input_Tab_configuration.png

Are you using a proxy on your internet connection? If so you will need to configure the properties on the node's Proxy tab.

As the website is using HTTPS you may need to install the SSL certificate. The SSL guide is located here for a default install of the Lavastorm client:

C:\Program Files\Lavastorm\LAE6.1\docs\lal\SSL Connections Guide for Lavastorm.pdf

The data is not available from that web site as XML or JSON data. I would recommend using the Contact Us link on the page to ask whether the CTFC can make the data set available in an open data format in line with other Open Data initiatives.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: awilliams1024

You may want to try to isolate the issue by copying the HTTP node and configuring it to a HTTP site e.g. our home page: http://www.lavastorm.com/
If the node downloads the data then it would point to the cause being SSL-related. If you cannot connect then it would imply a routing issue.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: awilliams1024

Obviously, if the CTFC are not able to supply the data in XML/JSON format then CSV or Excel are alternatives, though not the preferred options.