Data360 Analyze

 View Only
Back to discussions
Expand all | Collapse all

How to read CSV files within a ZIP file

  • 1.  How to read CSV files within a ZIP file

    Posted 11-11-2024 00:08

    Hi Support

    I was hoping that someone could help me with reading csv files that are zipped without a password.

    Since our version of D360 is cloud based, I can only upload 6 files at a time, and I have over 1000+ files I would need to upload.  So, I was thinking that if I could zip them into one file and then use a node to read the zip file and then use the csv node to read the file contents.

    It would be great if someone could help by putting a simple graph together on how to read the zip file and I could then pull it apart and massage it for what I need.

    Looking forward to your replies



    ------------------------------
    andrew darnell
    Knowledge Community Shared Account
    ------------------------------


  • 2.  RE: How to read CSV files within a ZIP file

    Posted 11-11-2024 10:14
      |   view attached

    Extracting from a zip is pretty straight forward in python this will do it:

    import zipfile
with zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:
    zip_ref.extractall(directory_to_extract_to)

    I did a small flow that will delete the directory you want to extract to and recreated it, thus ensuring it is empty. Then extracting all contents of the zip so you can then read the directory to see what was in the zip file.


    Attachment(s)

    zip
    Extract Zip - 11 Nov 2024.zip   4 KB 1 version
    Original Message


  • 3.  RE: How to read CSV files within a ZIP file

    Posted 11-11-2024 17:47

    Hi Gerry

    Thanking You for your quick response, this works perfectly!!

    This has saved me so much time from manually uploading 1000's of files to just 1.zip

    Enjoy your day.



    ------------------------------
    andrew darnell
    Knowledge Community Shared Account
    ------------------------------

    Original Message


  • 4.  RE: How to read CSV files within a ZIP file

    Employee
    Posted 11-12-2024 05:29
      |   view attached

    I was working in parallel on a solution using an approach similar to Gerry's (thanks for the contribution) and thought I would post it in case it helps you or someone else.

    The aim was to produce a node that would allow you to use the Directory Picker and File Picker functionality to specify the location of the source zip (in particular when using a Server instance of Data360 Analyze where you may not have direct access to the underlying filesystem).

    Example extract CSV files from Zip file - node properties
    Unfortunately, there doesn't seem to be an easy way to allow the use the output value of the File Picker functionality within the Python script in a Transform node. The example node instead uses a two-step approach that enables you to use the Directory Picker to select the directory where you have uploaded the source zip file (in the green box above) and then use the File Picker to select the actual zip file (the blue box above).
    The example was built using Data360 v.3.14.0.
    Gerry - for some reason I was getting a file not found error with your node (I'm not sure why, I was using a Windows Server instance).


    ------------------------------
    Adrian Williams
    Precisely Software Inc.
    ------------------------------

    Attachment(s)

    zip
    Import_Zipped_CSV_Files - 12 Nov 2024.zip   7 KB 1 version
    Original Message


  • 5.  RE: How to read CSV files within a ZIP file

    Posted 11-12-2024 11:24

    The file is on my c drive. Did you edit it the ZipFile and ExtractDirectory properties?

     


    Original Message


  • 6.  RE: How to read CSV files within a ZIP file

    Employee
    Posted 11-12-2024 12:08

    Yes I updated them.



    ------------------------------
    Adrian Williams
    Precisely Software Inc.
    ------------------------------

    Original Message


  • 7.  RE: How to read CSV files within a ZIP file

    Posted 11-12-2024 12:33

    The only thing I can think of then is that I usually replace the \ with a / for all file names but I didn't in this instance for absolutely no particular reason. I think I saw before shutil.move() doesn't work with a backslash, I could be wrong just going off memory of the backslash causing problems for me before. Might be worth a try.


    Original Message


  • 8.  RE: How to read CSV files within a ZIP file

    Posted 29 days ago
    Edited by andrew darnell 29 days ago

    Hi Gerry

    For some reason I am unable to find the below screen to make further changes.

    When I click on the node>Properties>Configure I just get the below, am I missing some step?



    ------------------------------
    andrew darnell
    Knowledge Community Shared Account
    ------------------------------

    Original Message


  • 9.  RE: How to read CSV files within a ZIP file

    Employee
    Posted 28 days ago

    Andrew,

    I think what is displayed is dataflow properties, rather than node properties.  Click on the canvas, i.e. whitespace between the nodes, to deselect the nodes.  Then I think you'll see those properties.



    ------------------------------
    Ernest Jones
    Precisely Software Inc.
    PEARL RIVER NY
    ------------------------------

    Original Message


  • 10.  RE: How to read CSV files within a ZIP file

    Posted 26 days ago

    Hi Ernest

    Ah, that has done the trick.

    Thanking You



    ------------------------------
    andrew darnell
    Knowledge Community Shared Account
    ------------------------------

    Original Message


  • 11.  RE: How to read CSV files within a ZIP file

    Posted 11-12-2024 18:11

    Hi Gerry

    Thanking You for putting this graph together.

    I've downloaded this as well



    ------------------------------
    andrew darnell
    Knowledge Community Shared Account
    ------------------------------

    Original Message