Data360 DQ+

Infogix is currently implementing support for the ingestion and output of Apache Kafka messages in DQ+ and we are interested in better understanding our customer use cases.  Any feedback will help ensure our product meets your needs. 

• How is your organization moving towards using Apache Kafka for data streaming?
• What are some of the use cases?
• Are you looking to validate and reconcile messages in real-time or batch?

For those interested in Kafka, we also have a few new articles on the Infogix website:

- https://www.infogix.com/defending-data-quality-in-a-real-time-data-world/

- https://www.infogix.com/resources/data3sixty-for-kafka-data-sheet/

My organization has a new administrative system that exports data using Kafka. We'd like to be able to directly consume these Kafka topics/streams in DQ+ to do things like data quality checks. Until this functionality appears in DQ+ we'll  probably need to stream data to a file data source which will be read and processed on a scheduled basis. A couple of features that seem like they would be needed are:

• Ability to trigger execution of an analysis or process model directly from the Kafka topic.
• Ability to control how often the analysis or process model is run: once for each message, only after a specified number of messages have accumulated, on a timed basis, or some combination of all of these. 

However, we don't have a well-developed set of requirements right now other than the need to ingest data from Kafka. 

Rob,

Thanks for the post.  We currently have the capabilities you describe available in tech preview mode in the on premise version of the solution.  Most of our customers are in the early stages of gathering requirements for Kafka. As more details become available we will be sharing them on the site.

Thanks.