Data360 Analyze

Hi, assume i have the following problem:

I have a sharepoint directory, containing excel files. I want to capture the changes in the directory, as in files being removed, added, files changing in size. And after that i want to upload it to an ADLS with the Analyze nodes(Sharepoint and ADLS nodes).

I would like to capture changes each 5 minutes, i know i can build this solution in Analyze with a while loop that checks for changes each 5 minutes in the Sharepoint directory, but then i have to have this schedule running 24/7, and it doesn't seem as convenient.

Is there a way i can make this more trigger based? That i only run my schedule once a change has been made in the Sharepoint Directory?

Best regards

I'm not aware of any trigger mechanism for changes to SharePoint that would be able to initiate the running of an Analyze data flow.

You could have a schedule that is set to run every 5 minutes that executes a data flow which checks for any changes to the directory's contents. You would need a mechanism to persist the previous state of the directory (e.g. in a database). This approach is more transparent than a continuously running data flow (in that the schedule runs show when the checks were made). However, it is still polling for changes.

Note that all updates would need to be completed within the scheduler's periodicity (5 minutes) else the next scheduled run would not start (and this would be indicated in the Runs list for the schedule). The next scheduled run after the previous run completed would then be executed.