LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

Node Name: GZip Extract
Node Category: Acquisition
Use Case: Consider using this node when attempting to access data that exists inside of a gzip file
Tested on Release: 4.1.6
External Dependencies: None
Practical Usage Example: An example use of this node has been provided in the prototype examples graph which is released in conjunction with the prototype library. Please see that graph for a executable use case.
Node Documentation:

Functional Description: Parameters:

• FileName: To extract a single archive, enter the full path and file name here.
• FileNameExpr:To extract multiple archives, enter the name of the input field containing the file paths here. Should be used in conjunction with a Directory List node or similar
• ArchivePath:Path into which extracted files should be placed
• DeleteSourceFile:If true, the original archive file will be deleted after its contents have been extracted.

Known Limitations: None

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mgajdosik

Hi,

I got following error, when trying to extract huge file, not sure what the size was exactly and unfortunately cannot find it anymore. Is there something that needs to be set up on the server to handle larger files? Or the node needs to be fixed?

Thanks
Marek


Got Python exception
Installing log (09/26/2010 08:51:37) ...
2010-09-26 08:52:20; Level: 3; Type: 0; Desc: ""; File: "/opt/lavastorm/python/lib/python2.4/gzip.py"; Line: 295; Context: "_add_read_data( ) - Exception"
2010-09-26 08:52:20; Level: 3; Type: 0; Desc: "Error pumping node (Traceback (most recent call last):
File "/opt/lavastorm/lib/python/lavastorm/brain/python1.py", line 912, in Run
while n.pump(quant):
File "/opt/lavastorm/db/../tmp/brain/Ziggo_E2E_Kenan/20100912/temp.gn-furap-prd-01.25465.7677.1285483897387.prop", line 48, in pump
outfile.write(zip.read())
File "/opt/lavastorm/python/lib/python2.4/gzip.py", line 218, in read
self._read(readsize)
File "/opt/lavastorm/python/lib/python2.4/gzip.py", line 278, in _read
self._add_read_data( uncompress )
File "/opt/lavastorm/python/lib/python2.4/gzip.py", line 295, in _add_read_data
self.extrabuf = self.extrabuf + data
MemoryError
)"; File: "/opt/lavastorm/python/lib/python2.4/gzip.py"; Line: 295; Context: "_add_read_data( ) - Exception"
Uninstalling log (09/26/2010 08:52:20) ...
Error in sys.excepthook:Traceback (most recent call last): File "/opt/lavastorm/lib/python/lavastorm/brain/python1.py", line 947, in <lambda> return lambda exc, value, tb: Python1ExceptHook(control, exc, value, tb)TypeError: 'NoneType' object is not callableOriginal exception was:Traceback (most recent call last): File "/opt/lavastorm/db/../tmp/brain/Ziggo_E2E_Kenan/20100912/temp.gn-furap-prd-01.25465.7676.1285483897386.prop", line 1, in ? import lavastorm.brain.python1 File "/opt/lavastorm/lib/python/lavastorm/brain/python1.py", line 956, in ? runMain() File "/opt/lavastorm/lib/python/lavastorm/brain/python1.py", line 954, in runMain raise BrainNodeException, "Encountered error running node"lavastorm.brain.python1.BrainNodeException: Encountered error running node

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mmarinelli

As it was originally implemented, this node has no memory management functionality, so I'm assuming that your problem occurs because the Python gzip library is trying to load the entire source file into memory as it expands it. I've attached a node which implements a buffering mechanism which I think should solve the problem. I set the buffer size to 1MiB by default, but I'm not sure what the optimal performance setting should be.

Please try this node and let me know if it works. If this fixes the problem, I will update the prototype library with this fix for all the archiving nodes.
Attachments:
GZip Extract.brn

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mgajdosik

Hi,

I found another issue. Well it is caused by incorrect use, but I think that this can be handled. If I am trying to extract other than gzipped file, then the file created will be empty (0 bytes long). If this extraction occurs to same directory then the original file got removed. Could there be fix implemented which would check if gzip is really gzip (regardless what extension is) and then if not then either log or error it or whatever. Especially when we have delete original file funcionality set to true and we unfortunately brough as input some files, which are not gzipped by wrong filtering, then there is a problem.

Thanks

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mgajdosik

Thanks for fixing issue with extraction of huge files. I just tested it out on the same file as I got error on before and it worked.

Thanks