LAE

Welcome to the LAE community!  Please feel free to start a discussion in the discussion tab or join in a conversation.

Discussions

Members

Resources

Events

 View Only
Back to discussions
Expand all | Collapse all

File import via HTTP transfer

  • 1.  File import via HTTP transfer

    Employee
    Posted 10-11-2011 14:27

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: SteveAkers

    Does anyone out there have any nodes developed to do directory and file transfer for HTTP?

    I have a number of .csv files posted on a server accessible via http. Currently I navigate via browser to get a directory listing and then click on each individual file and "save as" and then use the standard directory and csv nodes to get them into my graph for analysis.

    The ideal solution would be to have a HTTP directory node and HTTP file transfer node so that I don't have to go through the recursive manual process of "save as".


  • 2.  RE: File import via HTTP transfer

    Employee
    Posted 10-13-2011 02:04

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: erno

    Could you just use wget to download files to your computer first? You could even automate the process using Scheduled Tasks -feature in Windows (Control Panel).

    wget -nd -r --accept=csv,txt -l 1 www.xyz.org/datadir


  • 3.  RE: File import via HTTP transfer

    Employee
    Posted 10-13-2011 11:59

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: stonysmith

    Attached is a node that I wrote some time ago. You'll need to change how rows 31-36 process the data if you are reading a binary file such as an Excel document.
    Attachments:
    ReadWebpage.brn


  • 4.  RE: File import via HTTP transfer

    Employee
    Posted 10-27-2011 10:35

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: erno

    Originally posted by: stonysmith
					

    Attached is a node that I wrote some time ago.  You'll need to change how rows 31-36 process the data if you are reading a binary file such as an Excel document.
    Thanks, the node works great!

    I have one further question though: Would it be possible to fetch XML data with the node and pass data straight to XMLpy-node somehow? I'm aiming to parse some 100k exchange rates from http://www.ecb.europa.eu/stats/eurof...fxref-hist.xml and would find XMLpy useful here.


  • 5.  RE: File import via HTTP transfer

    Employee
    Posted 10-31-2011 08:58

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: rpigneri

    Dear Erno,

    All Acquisition nodes are built to bring a file into the LAE environment from the file system. Because of this, there is no way to pass XML data into the XMLpy File node via an input BRD. Instead, the Read WebPage node can write the file to disk and then output the full path of the file it created to its output pin. The XMLpy File node is built to take an input filename from an input BRD via the FilenameExpr parameter.

    Feel free to write back if you have any difficulties.

    Rocco


  • 6.  RE: File import via HTTP transfer

    Employee
    Posted 11-03-2011 03:21

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: erno

    Originally posted by: rpigneri
					

    Dear Erno,

    
All Acquisition nodes are built to bring a file into the LAE environment from the file system.  Because of this, there is no way to pass XML data into the XMLpy File node via an input BRD.  Instead, the Read WebPage node can write the file to disk and then output the full path of the file it created to its output pin.  The XMLpy File node is built to take an input filename from an input BRD via the FilenameExpr parameter.

    
Feel free to write back if you have any difficulties.

    
Rocco
    Could you give me an example how to write to a file directly from Read WebPage node? With BRD output I'm able to get almost perfect results, but it adds htmlpage:string -line to the beginning of the ouput, which messes XMLpy.
    Attachments:
    eurofx.brg


  • 7.  RE: File import via HTTP transfer

    Employee
    Posted 11-11-2011 08:16

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: stonysmith

    Originally posted by: erno
					

    Could you give me an example how to write to a file directly from Read WebPage node? With BRD output I'm able to get almost perfect results, but it adds htmlpage:string -line to the beginning of the ouput, which messes XMLpy.
    Attached is a graph that shows how to use OutputDelimited to write the file.

    To get OuptutDelimted to work:
    - specify FieldNames=None
    - specify some value for QuoteCharacter

    The tricky part here is to get OutputDelimited to work right, you have to specify a QuoteCharacter that does not exist in the input data. It might take some experimenting to find the right character to use. In this case we used the backslash, but we also got it to work with � .

    I would be interested to see what you use for the XMLpy node. I'm not very familiar with it, and I'm not sure how it should be configured to read this particular XML.
    Attachments:
    euro.brg


  • 8.  RE: File import via HTTP transfer

    Employee
    Posted 11-12-2011 00:05

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: erno

    I feel awfully stupid for not figuring this out before. Thank you so much, got my exchange rate fetch working!


  • 9.  RE: File import via HTTP transfer

    Employee
    Posted 10-24-2013 08:18

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: kaktusz65

    Dear All

    I tried to use the attached euro.brg sample file, but I received :
    python errno 10061 no connection could be made because the target machine actively refused it

    What can be the problem ?

    Best regards

    Zoltan


  • 10.  RE: File import via HTTP transfer

    Employee
    Posted 10-27-2013 06:34

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: Tim Meagher

    Hi,

    I'm not sure about that specific node or graph, because it was just posted here on the forums without any Lavastorm support.
    However, since then, the HTTP node has been released with LAL.
    So if you are trying to make a HTTP request, then you should just be able to get the latest LAL release and use the HTTP node with the parameters set as shown in the attached example BRG.

    Then, you can either just save off the XML using an output BRD, or pass it through the Convert Tabular XML node in order to have the data fields returned converted to standard LAE data fields rather than an XML response chunk.

    Tim.
    Attachments:
    httpExample.brg


  • 11.  RE: File import via HTTP transfer

    Employee
    Posted 10-30-2013 11:41

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: kaktusz65

    Hi

    Thanks for the example. It works well from home, but inside the company network I still receive the error message. Do you have some recommendations for firewall or port settings, which port are used by Lavastorm ?

    Best regards

    Zoltan


  • 12.  RE: File import via HTTP transfer

    Employee
    Posted 10-30-2013 12:06

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: Tim Meagher

    Hey,
    Pretty sure since that's just standard HTTP it will be port 80.
    Bear in mind that the node is executed on the LAE server therefore the server would need to be able to send requests & receive responses on port 80.
    If you need to connect through a proxy server, there are parameters on the node that allow you to configure the proxy details.

    Regards,
    Tim.