LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mwillett

Hi there,

I expect I should know this one but haven't been able to work it out.

I'm importing a file with 30 fields (columns) of data. Number of records is not really relevant for this question. For each field, I want to run some basic data quality checks - these don't seem to exist in the Data Analyser but even if the examples I give do, I will possibly want to add others.

For example, I want to get the length of each record for each (and every) field and also know whether each record is all digits (again for each and every field). From that I will then want to group and agg this so I know the number of records with different string lengths and also the number of records that are all digits and those that are not - for each and every field.

I'm ok with the functions for the above but I'm keen to learn how to run these same DQ checks over every record and field of data in a more efficient way than creating a composite and passing in the field names as variables thirty different times.

Essentially some form of loop (assume I don't know the field names in advance to hard code something) seems required. Of course, it would also have to be such that when it loops to run on field #2 (and so on up to 30), it doesn't override the values for field #1 as I'd want to output results for all fields for further analysis.

Thanks,
Mike

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: e0055348

You can try something like this.

Basically use the list and map functions to apply tests over any number of columns. Then use the lookup and modifyFieldPrefix nodes to put it back together.
Attachments:
Column_tests.brg

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mwillett

Thanks, I've been off the forum for a couple of weeks but will look into this shortly.