LAE

Welcome to the LAE community!  Please feel free to start a discussion in the discussion tab or join in a conversation.

Discussions

Members

Resources

Events

 View Only

  • 1.  Unexpected Error - Debug

    Employee
    Posted 05-06-2015 06:35

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: jpstory

    Hi

    The following error comes up from time to time when I use the code below to extract zipped files, please could you help diagnose the issue, thanks!

    Here is the error message:
       <logEvent id="0" parent="0" chain="0" timestamp="1430918762000" level="ERROR">
      <message xsi:type="rawMessage"><![CDATA[Error pumping node (Traceback (most recent call last):
  File "C:\Program Files (x86)\Lavastorm\LAE 4.6\lib\python\lavastorm\brain\python1.py", line 917, in Run
    while n.pump(quant):
  File "C:\Program Files (x86)\Lavastorm\LAE 4.6\BRAINscratch\tmp\brain\My+Quarterly+Invoice\default\temp.A356D71ZWNSRPYT.9692.919.1430918567186.554930a321c260aa.prop", line 40, in pump
    success = self.processZip(filename)
  File "C:\Program Files (x86)\Lavastorm\LAE 4.6\BRAINscratch\tmp\brain\My+Quarterly+Invoice\default\temp.A356D71ZWNSRPYT.9692.919.1430918567186.554930a321c260aa.prop", line 80, in processZip
    raise braininfo.BrainNodeException, 'ERROR: ' + str(e)
BrainNodeException: ERROR: 
)]]></message>
      <detail xsi:type="cppDetail" context="processZip( ) - Exception" line="80" file="ls.brain.node.brainpython.python2implementationfile"      </detail>
   </logEvent>

Got Python exception

Node reported the following messages to stderr: 
Error in sys.excepthook:Traceback (most recent call last):  File "C:\Program Files (x86)\Lavastorm\LAE 4.6\lib\python\lavastorm\brain\python1.py", line 952, in <lambda>    r

    Here is the code:
    import braininfo
import os, zipfile

def setup(brainNodeControlObj, BrainNodeClass):
	class BrainNode(BrainNodeClass):
		def initialize(self):
			super(BrainNode, self).initialize()
			
			metadata = self.newMetadata()
			metadata.append("FileName", "String")
			self.outputs[0].metadata = metadata
			
			self.isMultifile = False
			if len(self.inputs) > 0:
				self.isMultifile = True
				self.FilenameField = self.properties.getString("ls.brain.node.ZipExtract.FileNameExpr")
			else:
				self.Filename = self.properties.getString("ls.brain.node.ZipExtract.FileName")
			
			self.extractPath = self.properties.getString("ls.brain.node.ZipExtract.ExtractToPath")
			self.deleteSourceFile = self.properties.getBool("ls.brain.node.ZipExtract.DeleteSourceFile")
			
			# Check if Extract Directory Exists (Create)
			if not os.path.exists(self.extractPath):
				os.makedirs(self.extractPath)
			
		def finalize(self, val):
			super(BrainNode, self).finalize(val)

		def pump(self, quant):
			while quant.permitsRunning(self):
				if self.isMultifile:
					inRec = self.inputs[0].read()
					
					if not inRec:
						return False
					
					# open Zip file
					filename = inRec[self.FilenameField]
					success = self.processZip(filename)
					
					# Delete Zip File if necessary
					if self.deleteSourceFile and success:
						os.remove(filename)
					
					return True
				else:
					# open the zip file
					filename = self.Filename
					success = self.processZip(filename)
					
					# Delete Zip File if necessary
					if self.deleteSourceFile and success:
						os.remove(self.Filename)
					
					return False
					
		def processZip(self, filename):
			success = False
			try:
				zip = zipfile.ZipFile(filename)
				for name in zip.namelist():
					if name.endswith('/') or name.endswith('\\'):
						os.mkdir(os.path.join(self.extractPath,name))
					else:
						outfilePath = os.path.join(self.extractPath,os.path.basename(name))
						outfile = open(outfilePath, 'wb')
						outfile.write(zip.read(name))
						outfile.close()
						outRec = self.outputs[0].newRecord()
						outRec["FileName"] = outfilePath
						self.outputs[0].write(outRec)
						success = True
				zip.close()
			except zipfile.error, e:
				success = False
				raise braininfo.BrainNodeException, 'ZIP ERROR: ' + str(e)
			except Exception, e:
				success = False
				raise braininfo.BrainNodeException, 'ERROR: ' + str(e)
	return BrainNode


  • 2.  RE: Unexpected Error - Debug

    Employee
    Posted 05-08-2015 06:12

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: jpstory

    Bumping

    Could you please take a look at this.
    I am using the code to unzip 100+ files with size of 500MB+ (in csv format), if the size is causing the problem how come the error is occurring on some of the files.

    I have tried unzip the files manually one by one and did not come across any issue ( files corrupted, unable to unzip etc.)


  • 3.  RE: Unexpected Error - Debug

    Employee
    Posted 05-11-2015 06:00

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: lnason

    Hi JP - I tried a test using your code above on a handful of files and didn't see any issues with it.

    When you say that you tried to unzip the files manually one by one, can you clarify the "manual" part a little? For example, were you using the unzip node to to unzip the files one by one?....or do you mean that you were using the Windows unzip utility (or something similar) to to unzip the files?

    Another question, does the error occur on the same files each time you try to run the unzip node? or does it seem to be more random in nature?

    It's hard to make any educated guess about what's going on based on the error message you included, so any other supporting information you can think of would be helpful in assisting you.

    Finally, one additional tip/request for this issue and other issues you post about in the future. Instead of posting the python code as you have above (which is good)...it's even more helpful if you include everything about the node by saving the node as a "brn" file. You can do this by following the steps below:

    1. Select the node in BRE and copy it (Edit->Copy or Ctrl+C)
    2. Paste the node into a text editor like Notepad
    3. Save the Notepad file with a ".brn" extension, such as PythonTest.brn (Also select "All Files" in the Save as Type picklist)
    4. Upload this file to the forum thread.

    You'll notice that you can take a ".brn" file and drag it from your desktop into the BRE canvas and the node will be added to the canvas. Posting the brn makes it easier for us to take a look at what you have configured. In some cases, the error might be caused by something you have configured elsewhere in the node (besides the python code itself).

    Cheers,
    Luke


  • 4.  RE: Unexpected Error - Debug

    Employee
    Posted 05-12-2015 13:22

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: jpstory

    Hi lnason

    Thanks for looking at this and apologies for just getting back to you as I was out for a few days.

    When I said extract the files manually, I meant using Windows unzip utility (right click on the zip file and select Extract All in the pop-up menu). I was trying to test if the zip files were corrupted so un-extractable no matter what.

    The issue occurred on the same files.

    The node setup is quite simple, only involves two nodes: one "Directory List" node and a "Zip Extract" node, here they are:
    Attachments:
    Zip Extract.brn


  • 5.  RE: Unexpected Error - Debug

    Employee
    Posted 05-13-2015 05:57

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: lnason

    JP - For a brn file, you actually have to copy the nodes themselves, by clicking on them...then cutting and pasting, as opposed to copying and pasting the code inside of a node.

    In any case, I ran a few more tests using the code you included and didn't run into any errors.

    Based on the information available (you can unzip some files, but not others), I have to assume that there is something amiss with the data in such a way that the Python zipfile module runs into an issue where Windows unzip does not.

    The only way I could troubleshoot this any further would be for you to provide the files you are using. I'm guessing this is probably sensitive information that you don't want to be passing around however.

    Luke