Being Right and Data Quality

Over the past 2 months I have found myself listening or participating in conversations that were driving for an absolute answer. I am by far no mystic guru or physocologist, but it is sad to see the energy we spend endeavoring to be right. Please don't take me wrong, this is not an excuse to be sloppy. But let's take a conversation like data quality. I witnessed a conversation the other day about some data that was of issue. It turns out a data dictionary and an understanding of the EDW was out of date due to some changes in an operating model. After reviewing the impact of this change is it flowed to a downstream system a straightforward way to deal with the issue was presented. This was well received by everyone and the conversation of changing it at source was avoided (as the data was still correct, but just needed to be interpreted differently). Fast forward a month. Some anomalies again started to raise there head. The data still seemed to be off. So some further analysis reconciled the issue, but then the ugly let's change it at source as other downstream consumption may be wrong. So let's not worry about the outcome in reality, let's review our options. 1. Change data at source. Always my preferred method if practical. In this case, the data at source was not an issue. So we could change the transformation rules that loaded the data, but now we are locked into enforcing a specific rule that will change again and cause an issue. Chances are this change would cost about $250k and may have other issues downstream. 2. Enforce some governance on the consumption of the data. We could share with all the possible downstream consumers of this data the insights found to see if they are effected. What would the cost be? In some cases small. In other cases big. ,syne now we have to change multiple downstream jobs. Or maybe the required change does not effect the other groups and they need the data as it is. 3. Do nothing. Is there a way to quantify the impact? Is it worth the money and continued delay. I had a great Cleint who could boil a situation down to how many customers and how much money. He saved his organization millions of dollars not chasing down what were small anomalies with limited impact. So happily I believe this comes down to proper governance and context. How would you handle is issue?

Comments

[MBT Sale]

Good stuff as per usual, thanks. I do hope this kind of thing gets more exposure.

[MBT Trainers]

Thanks for taking this opportunity to discuss this, I feel fervently about this and I like learning about this subject.

[Herve Leger Dress]

Thanks, ibm laptops. I found the questions really helped. Some of my answers showed me some things I need to work on.