Thursday, December 15, 2016

Data Governance with CloverETL

Let's start by defining data governance. According to wikipedia (because wikipedia knows all), "Data governance is a control that ensures that the data entry by an operations team member or by an automated process meets precise standards, such as a business rule, a data definition and data integrity constraints in the data model. " So at the end of the day, an organization's data governance policy is specifically created to standardize the data model to improve operational efficiency. If you are given a project where you need to create a database schema, tables, columns, etc, you have your recipe in order to complete the task by following the data governance policy. However, the recipe is only as good as the cook who is creating it. 

Data governance isn't a 'glory' topic. In fact, it's quite the opposite! Developers (database, application, etc) and DevOps absolutely hate projects that involve implementing or following an organizations data governance policy. Why? It's time consuming and some standards they may not agree with (to name a couple). Would you rather have the policy up on your monitor while you are creating a new data feed or create the feed as quickly as possible? I would venture a guess that you go with the latter because it's more realistic in today's pace of business. What if you work for an organization where it's required by law? Some data governance policies are put in place to satisfy regulatory laws. Think about financial companies. We need a policy for maintaining data for banks, lending, and stock markets for handling sensitive data. 

Have you thought about trying to implement a data governance policy and failed? Perhaps you implemented your first data governance policy 5 years ago. I bet you that the policy isn't being followed for all of your data systems. There are decisions made everyday that break data governance policies and almost no way to detect these small violations. I have a perfect solution that you never would have thought of. Use CloverETL to define your data governance policy! Most people think of CloverETL as a data integration software tool and not something that CloverETL could handle. But think of this this way. Your data governance policy is the set of business rules that you create graphs for and the data that you are validating is actually the structure and syntax of your data systems. 

You can use CloverETL to validate all database schemas that were created on your operational systems and then report data governance violations for the structures. Better yet, run the solution on the development and QA environments so that you catch the governance violations before they even touch production. So what can CloverETL do for you? 

-Check all syntax for data systems including: database naming conventions and data types. 
-Validate the structure of your data based on your business rules for data. 
-Verify and report on user access to databases for each of your operational databases. CloverETL can -Monitor the quality of your data. 

The solution can run daily, weekly, monthly, yearly or any other interval you deem relevant. This is a great use case and policy to automate because once you spend the time upfront automating the process, you can enforce the policy without much effort. If you have any questions about data governance with CloverETL, please don't hesitate to reach out. 





No comments:

Post a Comment