LAE

Welcome to the LAE community!  Please feel free to start a discussion in the discussion tab or join in a conversation.

Discussions

Members

Resources

Events

 View Only
  • 1.  Ftp

    Employee
    Posted 11-18-2009 21:37

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: PGeee

    is there a node or a trick to allow me to ftp a file from Lavstorm to an FTP / SFTP server

    Rgds

    PG


  • 2.  RE: Ftp

    Employee
    Posted 11-19-2009 12:06

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: timonk

    Good Day PG.

    While the Lavastorm softare does not come with a node that functions as you describe out of the box, it does however include in the CORE library a "Python" node. This node allows for users to create their own nodes to fit specific project or graph needs.

    For your convenience, I have pasted an example written by another user of how someone might use the Python node to implement an FTP node, with basic GET, PUT and LIST functions. Please feel free to use this as a guide, or as a solution if you feel it will help. Simply copy the text and paste it directly into the BRE editing pallette.

    Please understand however, that only the nodes and libraries we release with our product are supported. We will support the basic function of the Python node, such that it operates as intended (interfaces correctly with and runs against a server), but we cannot support any custom work that users may do with it.

    Please let us know if you have further questions.
    Regards
    Timon Koufopoulos,
    MDA Support.

    node follows below ###
    ####

    node:FTP
    bretype:core::Python
    editor:Label=FTP
    editor:sortkey=46cc36f059c86d97_2
    editor:icon=export.ico
    inclock:/=Generate_reports
    prop:Action=Get
    prop:Binary=true
    prop:LocalPath=C:\Documents and Settings\user\Desktop\brain data\worldseries.csv
    prop:Password=guest
    prop:Python2Implementation=<<EOX
    import braininfo
    import ftplib
    import sys
    import os.path

    def ftpgrablines(line='', lines=[]):
    if line:
    lines.append(line)
    else:
    result = lines[:]
    del lines[:]
    return result

    def setup(brainNodeControlObj, BrainNodeClass):
    class BrainNode(BrainNodeClass):

    def initialize(self):
    super(BrainNode, self).initialize()
    self.ftp = ftplib.FTP('{{^RemoteServer^}}', '{{^UserName^}}', '{{^Password^}}')
    self.logInfo("initialized ftplib")
    self.remote_path = '{{^RemotePath^}}'
    self.local_path = os.path.normpath(r'{{^LocalPath=.^}}')
    self.BinaryMode = '{{^Binary^}}'
    self.PutOrGet = '{{^Action^}}'

    if self.BinaryMode == "true":
    self.opMode = 'wb'
    else:
    self.opMode = 'w'

    def finalize(self, val):
    super(BrainNode, self).finalize(val)
    self.ftp.close()

    # This will handle the data being downloaded
    # It will be explained shortly
    def handleDownload(self, block):
    if self.BinaryMode == "true":
    self.f.write(block)
    else:
    self.f.write(block + "\n")

    def pump(self, quant):
    if self.PutOrGet == "Get":
    self.f = open(self.local_path, self.opMode)
    self.ftp.retrlines('LIST')
    if self.BinaryMode == "true":
    self.ftp.retrbinary('RETR ' + self.remote_path, self.handleDownload)
    else:
    self.ftp.retrlines('RETR ' + self.remote_path, self.handleDownload)
    self.f.close()

    if self.PutOrGet == "Put":
    self.f = open(self.local_path, 'r')
    if self.BinaryMode == "true":
    self.ftp.storbinary('STOR ' + self.remote_path, self.f)
    else:
    self.ftp.storlines('STOR ' + self.remote_path, self.f)
    self.f.close()

    if self.PutOrGet == "List":
    om = self.newMetadata()
    om.append("FileName","string")
    #om.append("Size","string")
    self.outputs[0].metadata = om
    self.ftp.retrlines("NLST "+self.remote_path, ftpgrablines)
    s = ftpgrablines()
    for line in s:
    rec = self.outputs[0].newRecord()
    #rec["Size"] = self.ftp.size(line)
    rec["FileName"] = line
    self.outputs[0].write(rec)

    return False
    return BrainNode
    EOX
    prop:RemotePath=/worldseries.csv
    prop:RemoteServer=ftp.lavastorm.com
    prop:UserName=guest
    editor:XY=750,60
    editor:propdef=RemoteServer|string|1||None
    editor:propdef=UserName|string|1||None
    editor:propdef=Password|password|1||None
    editor:propdef=Binary|boolean|1||None
    editor:propdef=Action|choice|1||Put|Get|List
    editor:propdef=RemotePath|string|1||None
    editor:propdef=LocalPath|filename|1||None
    editor:propoverride=Python2Implementation|2
    editor:propdoc=RemotePath=Remote path+filename.
    editor:propdoc=RemotePath=
    editor:propdoc=RemotePath=For PUT.. this is the name of the file to be written.
    editor:propdoc=RemotePath=
    editor:propdoc=RemotePath=For GET.. the name of file(s) to retrieve.
    editor:propdoc=RemotePath=Filenames of the form *.zip are allowed.
    editor:propdoc=RemotePath=If you use * then the LocalPath should be a folder name, not a single file name.
    end:FTP


  • 3.  RE: Ftp

    Employee
    Posted 04-30-2010 06:42

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: stonysmith

    Attached are three nodes.. A FTP_LIST, a FTP_GET, and a 'generic' form that supports all three: Get/List/Put

    The generic one is very close to the example above.

    Please note: Python does not support SFTP, but rather the simpler (nonsecure) FTP only.

    =========
    If anyone could suggest a method to retrieve the file size and date/time from the remote FTP server, I'd be glad to try to enhance the FTP_LIST node to support that feature.
    Attachments:
    ftp.brn


  • 4.  RE: Ftp

    Employee
    Posted 12-15-2010 11:47

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: burke

    Thanks for these examples, I can get the 'generic' node to PUT a single file, however I would like to be able to pass it a list or directory of files to ftp.

    I have a directory
    /path/to/my/local/files/
    and it contains
    subdir1/
    and
    subdir2/
    in each sub directory are files named such as
    2010-12-01_somefile.csv,
    2010-12-01_someotherfile.csv,
    2010-12-01_anotherfile.csv,
    2010-12-02_somefile.csv,
    2010-12-02_someotherfile.csv,
    2010-12-02_anotherfile.csv,
    2010-12-03_somefile.csv,
    2010-12-03_someotherfile.csv,
    2010-12-03_anotherfile.csv,
    etc
    I want to send them to
    /remote/server/subdir1/
    and
    /remote/server/subdir2/
    but I only want to send today's files from each directory to it's respective directory on the remote server i.e.
    PUT
    /path/to/my/local/files/subdir1/{{^CurrentDate^}}_somefile.csv
    as
    /remote/server/subdir1/{{^CurrentDate^}}_somefile.csv
    
    PUT
    /path/to/my/local/files/subdir1/{{^CurrentDate^}}_someotherfile.csv
    as
    /remote/server/subdir1/{{^CurrentDate^}}_someotherfile.csv
    
    
    PUT
    /path/to/my/local/files/subdir2/{{^CurrentDate^}}_somefile.csv
    as
    /remote/server/subdir2/{{^CurrentDate^}}_somefile.csv
    In a filter node I can get the variable/parameter expansion to work, however in the generic FTP node I can't get the expansion to work, instead the parameters I have defined for the {{^sub_directory^}} and {{^file_name^}} just pass the string of the input field name.

    I've also tried using
    /remote/server/subdir2/{{^CurrentDate^}}_*.csv
    but it only expands to 2010-12-15_*.csv so the * pattern is overlooked.

    Any help would be appreciated.


  • 5.  RE: Ftp

    Employee
    Posted 12-27-2010 10:00

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: rboccuzzi

    If I follow your requirements, can't you use a Directory List node to list the files on your local filesystem, using the pattern abilities and recursion of the Directory List.

    Take the output of that Directory List node, and use a filter node to determine the current date, and look for that string in the filename, filtering only those of today's date onward.

    Take the output of that, and put it to a split node, where you have one output for each of the source subdirs, and take each of those outputs, and connect it to the input of the ftp node found in the Lavastorm Labs library, where you can pass in the list of files to put, and just provide a directory to put them.

    I think this should all be fairly straightforward to wire together; let me know if you have any questions, or if I have missed something.

    Cheers
    Rich


  • 6.  RE: Ftp

    Employee
    Posted 01-04-2011 11:05

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: burke

    Firstly, thanks for the reply and Happy New Year!

    Originally posted by: rboccuzzi
    					

    If I follow your requirements, can't you use a Directory List node to list the files on your local filesystem, using the pattern abilities and recursion of the Directory List.
    Yes I am already doing that.

    Originally posted by: rboccuzzi
    					

    Take the output of that Directory List node, and use a filter node to determine the current date, and look for that string in the filename, filtering only those of today's date onward.
    Yes I am already doing that.

    Originally posted by: rboccuzzi
    					

    Take the output of that, and put it to a split node, where you have one output for each of the source subdirs, and take each of those outputs, and connect it to the input of the ftp node found in the Lavastorm Labs library, where you can pass in the list of files to put, and just provide a directory to put them.
    This is what I have tried, with the exception of "the ftp node found in the Lavastorm Labs library" I am using the nodes that stonysmith provided, I must not have the latest version of BRE (4.1.5.0 build 268) since I don't see an FTP node in the LSL library? I don't follow how the split node helps since I can already filter on the path of the file?

    Originally posted by: rboccuzzi
    					

    I think this should all be fairly straightforward to wire together; let me know if you have any questions, or if I have missed something.

    Cheers
    Rich
    Wiring it together was easy, passing it multiple files (or a directory) doesn't work though.


  • 7.  RE: Ftp

    Employee
    Posted 01-05-2011 10:49

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: rboccuzzi

    There is a prototype library that you can sign up to use, that is unsupported (except in the forums), and it has a node that would help you, a FTP Put that takes filenames as an input. You will need to request access to the Lavastorm Labs area of the forum, and then you will have access to download the prototype library. To request access, you need to go to your User CP, and select Group Membership, and request access from there.

    Cheers
    Rich


  • 8.  RE: Ftp

    Employee
    Posted 01-05-2011 11:31

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: burke

    Thanks, I have submitted the request.