LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dialpemo

Hi All,

I have been using the Http node to get information from different websites, I have had trouble downloading information where JavaScript is present, I have found that Node.js seems to work in this cases in combination with other modules, please see the following websites as references

http://www.smashingmagazine.com/2015...-google-search
http://code.tutsplus.com/tutorials/s...ejs--net-25560

my question is there of using these JavaScripts and the Modules mentioned in the above links in Lavastorm or do you know of a better way of web scraping websites that have Dynamic Content.

Kind Regards,

Alejandro Moncada

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: aop

You might want to try python libraries designed for web scraping as it would be easier to implement in LAE using the python node in LAE. I do not have personal experience of web scraping work with python so unfortunately I cannot give any library recommendations.

One obvious approach would also be to save development efforts and use a separate web scraping software to do the parsing and using the parsed data as input data for further processing in LAE.