Acryl Data has announced a new module for its data catalog platform. Acryl Observe will give users a single platform for data observability, governance, and discovery. Acryl Data is the main company behind the open source DataHub Project, the most widely-used open source metadata platform and data catalog and largest metadata community in the world.

The DataHub project is based on the DataHub metadata platform which was originally built at LinkedIn and subsequently open-sourced under the Apache 2.0 License. The project is described as a third generation metadata platform that enables data discovery, collaboration, governance, and end-to-end observability. Acryl Data is a major supporter of the project.

Acryl Observe is a new module for Acryl's platform. The developers say it is driven by a central metadata control plane, and continuously detects data quality incidents in real-time, automates anomaly detection to prevent breakages and drives faster resolution when they do occur.

The automatic detection of anomalies aims to prevent problems with data quality breakages, and to sort out problems more rapidly when something does go wrong. The developers say they propose data contracts for producers to review and accept, so that accountability is clearer.

The module offers a high-level view of the health of an organization’s data, so data managers can monitor the health of all datasets in real-time, with automated and user-defined assertions for data quality.

Pipelines and tasks can also be monitored in real-time, with automated checks on the health and analysis of downtime. The module can also be used to track what has been affected by incidents, so reducing time-to-restart on failed jobs. The dashboard can also be used to explore data tables to find active incidents, failing assertions, and "pending anomalies".

Acryl says the new module can also be used to build continuously-monitored SLAs for tasks, datasets, tables and columns, and enforce automated workflows that trigger when assertions fail.

Acryl Observe is available now on the Acryl Data platform.

More Information

Acryl Data Website

DataHub Project

Related Articles

Fiberplane Adds Metrics Tool

Google Builds Data Lake On BigQuery

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info