Data teams continue to face long-standing challenges: their customers often distrust their results, data providers frequently ignore their existence, and teams spend more time firefighting than creating insights. The demand for AI just makes it more complicated: no wonder many data teams experience PTSD.
The solution is simple: identify problems before they reach your customer. You need to implement data quality tests—lots of them. Check every table and column. See if anything is incorrect. Run these tests in production and incorporate them into the development process. Use the results to obtain data quality scores and implement improvements to your source systems.
Data teams work with hundreds or thousands of tables and often lack sufficient time to achieve data test coverage. That’s why, after decades of data engineering, we released an open-source tool that handles this for them.
DataKitchen’s open-source data quality and data observability tools aim to help data teams automatically generate 80% of the necessary data tests with just a few clicks, while providing an easy-to-use UI for collaborating on the remaining 20% of tests that are unique to their organization.