In this article, we delve into the often overlooked, but crucial aspect of data quality – data lineage. Data lineage records the flow of data and all the transformations throughout its life-cycle, from source to destination. Understanding this is vital...
-In this blog, we explore how to ensure data quality in a Spark Scala ETL (Extract, Transform, Load) job. To achieve this, we leverage Deequ, an open-source library, to define and enforce various data quality checks. If you need a...
-We’ve all heard the phrase “garbage in, garbage out” which highlights the importance of quality data for data-driven systems. Here, quality data can be interpreted in two ways: firstly as clean and well-standardized data that meets expectations, and secondly, as...
-“Data is eating the world”, is a well-known variant on Mark Andreesen’s famous quote “Software is eating the world”. In our current output driven society everything is measured by an interpretation of a set of data, even by the people...