Data360 Analyze

I am trying to use the new 3.1.5 version to use a python script to calculate the last x amount of days from today, I basically want a script to determine the last x amount of days via a python script and then filter out only those records. I need to use a script and not the filters, because it will vary from Last 7, Last 28, Last Year. 

I am thinking it involves importing datetime and then subtracting a time delta, but the node seems to be more complicated now that python needs to be configured and I can't even figure out how to start with the script because it looks like it requires 3 different scripts.

Can someone at Lavastorm do a video example of how to work with the PYTHON nodes now? It looks like a multi step process to even get a script to work. Thank you.

Hi Nancy,

Your approach is correct. I just tried the following and it worked for me:

In ConfigureFields field of the Transform node I put the following:
---

from datetime import datetime, timedelta

N = 2

date_N_days_ago = datetime.now() - timedelta(days=N)
---
I can then use the date_N_days_ago value in scripts in ProcessRecords and ConfigureFields. For example to output the date_N_days_ago value as a field I could do the following:

ConfigureFields:
----

from datetime import datetime, timedelta

#define a new field of type datetime on output out1
out1.Date_N_Days_Ago = datetime

N = 2

date_N_days_ago = datetime.now() - timedelta(days=N)
---
ProcessRecords
---
#output the value of date_N_days_ago in field out1.Date_N_Days_Ago

out1.Date_N_Days_Ago = date_N_days_ago
---
Let me know if this helps.

Mark

Nancy,

If you want to suppress records in a transform node you need to set the output to None based on the condition you have identified. The example below based on Mark's above should give you all the information you need.

Step 1

Create a Create Data node with the following content in the Data field

id:int,create:datetime
1,2017-08-01 12:35:41
2,2017-09-01 02:00:00
3,2017-07-01 05:23:21
4,2017-04-17 18:12:21
5,2017-06-21 15:12:21

Step 2

Add a Transform node with the following in ConfigureFields

from datetime import datetime, timedelta

out1 += in1

N = 100

date_N_days_ago = datetime.now() - timedelta(days=N)

And the following in ProcessRecords

out1 += in1

if in1.create < date_N_days_ago:
    out1 = None

Note the result will only output those records newer than 100 days.

Regards, Tim.

Hi Nancy,

Did you have a chance to look at what we suggested?  Did this give you what you wanted?

Thanks,

Catharine

Hello, thanks for the tips. 

This example works as reproduced but it's not working with my data. I have a datetime field but I am trying to filter off a converted date to string. So that's more than likely my problem,

So ideally I would want my filter to find the x number of days then filter off the new truncated date field I created. 

When I output the datetime though I don't want to see the entire datetime format, just a truncated version of it. 

Hi Nancy,

The data flow attached to this knowledgebase article provides some examples of extracting elements of a date object as a string. The example data in the article used a date object but the same code can be used to extract the corresponding elements from a datetime object.

There are a number of articles describing the use of Python within the Transform node in the Tips & Tricks section of the knowledge base.

Regards,

Adrian

Hi Nancy,

 If you can provide an example or two of what the data looks like in the string, that will help.

Can you also clarify what you mean by these statements (with an example):

So ideally I would want my filter to find the x number of days then filter off the new truncated date field I created. 

     -  What does the new truncated date field look like?

When I output the datetime though I don't want to see the entire datetime format, just a truncated version of it. 

    -  So you want it to look like the new truncated date field you mentioned in your previous statement?

Hopefully this will give us enough information to be able to help you accomplish what you want.

Thanks,

Catharine