Basics of DMFs
DMFs are functions that return a specific metric, such as the number of NULL values in a column, the number of unique values, or the freshness of a table. The system DMFs provided by Snowflake in the SNOWFLAKE.CORE schema cover many standard requirements and enable a quick start without additional development effort. For more complex or domain-specific use cases, you can create custom DMFs and store them flexibly in the desired database and schema.
Once a DMF is assigned to a table or view, Snowflake handles the automatic execution. The metrics are calculated at defined intervals—for example, several times a day or aligned with the respective data load cycle. All results are stored centrally in a dedicated event table, creating a consistent and traceable history of data quality. This approach not only makes data quality visible but also allows for comparison over time and across different data objects.
DMFs can be applied to a wide variety of objects, including tables, views, materialized views, dynamic tables, external tables, and Apache Iceberg tables. As a result, the approach is suitable for both traditional analytics workloads and modern data architectures. At the same time, there are intentionally set limits—such as on the maximum number of DMF mappings or on usage in shared or trial accounts—that should be taken into account during planning.
Added Value and Potential Applications of DMFs
DMFs enable the continuous measurement of quality metrics such as completeness, uniqueness, and timeliness, rather than checking them only sporadically or manually. This allows for the automation of error-correction measures and ensures high-quality data on an ongoing basis, which in turn strengthens all stakeholders’ trust in this data. To this end, Snowflake provides both predefined, system-native DMFs and the ability to define custom metrics that are precisely tailored to business or technical requirements.
Data Quality and DMFs are available as a feature of Snowflake Enterprise Edition. One advantage of the Snowflake approach lies in its technical implementation. DMFs utilize serverless compute resources that are fully managed by Snowflake. There is no need to provision or manage additional virtual warehouses, and billing is based solely on the scheduled execution of the metrics. Creating DMFs is free of charge, and ad-hoc calls—such as those made for analysis—also incur no costs. Usage is transparently reported under “Data Quality Monitoring” and can be tracked in detail.
Practical Application of DMFs
An example implementation scenario in an e-commerce business illustrates what such a data quality workflow looks like in practice.
Specifically, the example implements the following quality rules, among others:
- Uniqueness: CUSTOMER_IDs must not occur more than once
- Timeliness: Customer data may be no more than 10 days old
- Business plausibility: Order totals must not be negative
First, a central schedule for Data Metric Functions is set up, which defines the frequency of the quality checks. Based on this, system-specific DMFs—such as those for detecting NULL values, checking uniqueness, or measuring data recency—are specifically linked to the customer data table. This ensures that customers can be uniquely identified by their CUSTOMER_ID and that, for example, their records are no older than 10 days. In addition, a user-defined DMF is used to map a business-specific quality requirement. This DMF checks the order total (TOTAL_AMOUNT) in an order table for negative values. This ensures, for example, that no unintended discount combinations result in negative order totals and that the company does not mistakenly pay out money.
A crucial aspect here is that a DMF initially only measures a value but does not yet make a determination as to whether it is acceptable. To turn a measurement into a true quality check, so-called “expectations” are used. These define which range of values is considered “good.” The result of the DMF is automatically compared to this expectation and evaluated as pass or fail. In this way, formal and reproducible data quality rules can be established without having to build logic outside of Snowflake.
The example shown illustrates how DMFs are systematically registered, automatically executed, and their results made centrally available. This takes place entirely within Snowflake and without any additional infrastructure.
From Measurement to Active Monitoring
When combined with alerts, data quality ultimately becomes an active monitoring mechanism. In the event of deviations or threshold violations, automated notifications can be triggered, allowing problems to be detected and resolved early on. Various roles within the company benefit equally from this: data stewards gain transparency into the current state of their data, data engineers can respond quickly and effectively, and platform administrators can keep an eye on costs, performance, and consistency. Data quality thus evolves from a reactive task into an integral part of data governance.
Conclusion
Overall, Data Metric Functions demonstrate how data quality can be directly embedded within the data platform. Instead of relying on external tools, additional infrastructure, or manual verification processes, quality is measured right where the data is generated and processed. This enables companies to establish a reliable foundation for compliance, stable SLAs, and, above all, trust in their data. Anyone who wants to make data-driven decisions in the long term will find it hard to do without a structured, automated data quality approach like the DMFs in Snowflake.
– – – – –
Related Links
👉 www.adesso.de
Photo: unsplash
