Project X Discussions

Data Governance Part 9 Data Quality Story

About a year and a half ago my wife took out a credit card from a major US department store to obtain a big discount on a major purchase.  Although there were a few wrinkles in paying the bill. it worked out fine.   She never used the card again but about two months ago we got a bill from them for a store we had never been in.  I called up to inquire and the customer service person looked things up and said it was clearly not her signature and she would refer it to the fraud department but it would take a while.

About a month later we got another bill with delinquency charges on the bill.  We called up again and was connected with the fraud group.  They said it was not fraud it was a mistake.  A person was at the store in this place and did not have the card so gave the last four digits of the social security number and an account popped up.  As background, we are from Canada and we have a Social Insurance Number and my wife had to give this to get her credit card.   The last four digits on my wife's card was the same as this person''s four digits on the social security number.   Eventually it was all cleared up but I am willing to bet this problem will affect my wife's credit rating.

So here we have a data quality issue which becomes a business issue because we will think twice about shopping in that store again.  How many of these customer service problem are causing people to go somewhere else to shop?  The most precious commodity of every business is their customers and how quickly they lose our trust.

Another data security issue is that, in Canada, a business cannot use our Social Insurance Number as a means of identifying us.  The only people who can use it are the tax people and the social security people.   Interesting how anybody can ask for your SSN in the United States.  

I wonder how often in all our businesses, data quality becomes a customer service issue.   

Data Governance Part 8 Is Data Quality an IT Issue?

I just read a very provocative blog about whether data quality is an  IT issue and the author has a good point.  See http://www.dataqualitypro.com/data-quality-forum/post/667593

Of course data quality is a business issue but that does not mean that IT is not intimately involved.  Often the business rules specified by the business are not impemented correctly and sometimes the data validation routines are incorrect.  There are numerous other examples, but in the end the business must make sure that these errors are corrected by IT.  Often these errors do not show up during testing but later when the application is being used.

Often the phrase that data quality is a business issue is interpreted as IT not acknowledging their role.  Nothing could be further from the truth and the business must work with IT to correct the data quality issue.  Often the responsibility is shared and must be resolved by the groups working together to resolve some very complex data definitions and business rules.  Effective data governance is an effective team effort with everybody working together to deal with issues rather than placing blame or pointing fingers.

Data Governance Part 7 Data Stewardship

One of the key ingredients in data governance is the concept of data stewardship.  Every project should be familiar with the concept and introduce the concept on all their projects very early.  I think it is a great way to get started on data governance and will get great support from the business owners of the data. 

I must give Robert Seiner some credit for these ideas because has written some excellent material on this subject.  You can see his work at www.tdan.com

Data Stewardship is the formalization of accountability for the management of data resources.

I have asked several people on various projects recently if they knew who the owner of various data resources was on their projects and with a little thought each one said yes.  These people are the informal data stewards and our thesis is that they should be recognized formally and explained their role.  A role they are already fulfilling on an informal basis.

The next step is to create metadata for the data steward which contains the names of the stewards and the data they steward.  Seiner calls this the Stewardship Repository.  In addition the responsibility should be written in their job description.  This action simply formalizes an activity they already do.

The great news is that we now know the answer to the question "Who owns the data?"  We then know who to go to if we have questions about the data and its status.  I bet everybody in the data warehousing business would welcome a place to go to find out who owns what data. 

My suggestions is that early on in every project the Business Analyst be assigned the responsibility of adding to the Stewardship Repository.  I am sure its existence would save the Business Analysts a significant of time.

This is the first step in data stewardship but a really important and relatively simple step.  I invite your comments. 

Collaboration is More Than Just Talk

As most of you who follow our blog know, Project X's RapidBI Approach and Platform is linked and depends very much on Agile Development and Real Collaboration.

The challenge with the word collaboration is that often it is just talk.  I have had the opportunity to work with someone who knew how to collaborate.  He is also a Lean Six Sigma guy and that doesn't hurt either.  It was a real pleasure.  We had some bumps coming up with common frameworks, we are both type A personalities and that was often fun.

But after about 1 week I really felt he was a collaborative partner.  We got to a comfort level I felt that allowed for often open or frank discussion that hopefully he would agree that neither of us used to our advantage when things got tough.

Exit that phase in the project/program and we had new people come in.  Sadly we can not get past the 'Just Talk' phase of collaboration.  This is tell me what I need to do, I will do it and then throw whatever back at you.  Sadly not even jumping back onto our collaboration platform helped us break through.  I just did an audit of the collaboration environment and there is nothing there.

Funny how we are spinning.  Less talk, Way less oversight meetings and lets get together and do it.

Data Governance Part 6 The Role of IT

I would like to discuss now the role of the IT folks in the data governance process.  In my previous blogs, I have emphasized how important it is that the process be business led.  However the IT folks have a very important role in data governance.  The role is to provide assistance and support in implementing the decision of the governing body.  In addition the IT folks have a responsibility to raise issues that require decisions.

One of the key responsibilities of the IT folks and the business is to maintain an open dialog on issues.  The challenge for the IT folks is to express the issues in plain language as opposed to talking to IT speak.  IT folks have a great tendency to talk in slang and short forms because they get tired of repeating the same words over and over again.  I think it is the responsibility of the business people to call them on this whenever it is used. 

In addition to the obvious reason of understanding each other, sometimes the IT folks often do not know the meanings of some of the terms they use.  Another thing is that some words lose their meaning through overuse.  For example, the word "system" really does not have a very specific meaning any more.  Let me give you other examples: program, software, web 1.0, web 2.0, SOA, and on and on.  However often the business people do the same thing.  Some of the worst offender are the defense, the aerospace and telecommunication industries.  These groups use so many short forms one would swear they are speaking another language. 

I am emphasizing this understanding as being important because I think it is key for effective data governance.   I will continue developing ideas on governance in future blogs. 

Microstrategy Security - Part 1

There have been a lot of questions to me around what kind of security can be set in MicroStrategy.  One of the most requested is allowing access to certain users to specific reports.  The answer is Yes we can limit access to reports based on user/user groups.  The type of user/license also will determine access to objects.  For example if I have an Architect license in development I can modify all Schema as well as Application objects.  If the license is a desktop developer license they would not have access to change Schema objects.

My experience is more around setting up security at the user and application layer.  Security filters and access list allow the users to view the data they are authorized to review and at a specific level.

I looked on the MicroStrategy site to see if they had any good information on security.  Its high level but explains the different levels:

There are four main architectural layers that must be completely secure: user layer, transmission layer, application layer and data layer.

• User layer: Users must be authenticated and authorized to access data.
• Transmission layer: Data transmission must be secured and hacker-proof.
• Application layer: Only appropriate application functionality should be provided to users with associated privileges.
• Data layer: Users must have access only to data for which they have appropriate privileges.

MicroStrategy's security architecture implements tight security at every layer.

Robust Security Functionality: To ensure robust security, MicroStrategy implements industry-leading security measures at every layer of the architecture.

Some examples for each layer include:

• User layer: integration with NT and LDAP security.
• Transmission layer: 128-bit SSL transmission, 128-bit encryption of data, multiple firewall (DMZ) configuration with no database connection on web server.
• Application layer: 3-tier security system encompassing privileges, access control and security profiles.
• Data layer: security filters and database views, revolutionary connection mapping."

My experience is more around setting up security at the user and application layer.  Security filters and access list allow the users to view the data they are authorized to review and at a specific level.

CoffeeCAST - #53 - BI Governance

Michael and I sat down after a great morning of review on lessons learned and wanted to quickly chat about Business Intelligence Governance.

In today's CoffeeCAST we talk about the main four areas of focus without going too deep on any one.  They are:

1. BI Governance committee
2. BI Lifecycle / Process
3. End-User Support Structure
4. BI Program Review

53 CoffeeCAST #53 - BI Governance

This was a broad and interesting area.  I think next week I will sit down again and talk specifics on these areas as a single discussion.

This is just a starting point to the conversation, so feel free to join in and add your thoughts.

Data Goverence Part 5 A Fairy Tale

This is an interesting fairy tale about one aspect of data goverence.   Have a look at http://youtube.com/watch?v=TbzQvswrOTw

I think that data quality is a dragon we need to tame and bring into the service of our organizations.    I think the knights in his battle are the Data Stewards which I will write more about in subsequent blogs. 

To give you a clue, the data stewards actually exist in your organizations and are not properly recognized.  

Comments invited. 

Data Governance Part 4 Readiness

I would like to encourage people who are striving to introduce data governance into their organization to have patience, persistence and perseverance.  Most organizations are not ready to embrace all the aspects of data governance at the beginning but as the various elements of governance are introduced to address business issues, readiness will increase.

I define readiness as the things people are ready and willing to do.  I have learned through painful experience that people will only do what they are ready to do. 

As we introduce change, and the change is successful, people's readiness will increase.  As people become more confident, their readiness increases.  As they gain more support and recognition from others in the organization, their readiness increases.  As they get specific feedback from their boss and peers, readiness increases.  Business pressures will often increase people’s readiness to try new approaches. 

Readiness can be increased by pressure from senior management to adopt new approaches. 

Simple stated, data governance is the readiness of the organization to take ownership of the stewardship & management of the processes around data.  The ownership will have to be shared with the IT folks who are very protective of the data.  I think the IT folks are very good at implementing processes once they are defined.  The processes need to built to serve the business, not the IT organization.  A strong business leader who understands the power of information can be a great ally of data governance.  These leaders need our support and cooperation. 

Many times the business leaders need to see how information can help them transform the organization.  With patience, persistence and perseverance we can help them transform the organization.  They need support from the IT organization because the implementation requires work by the data warehousing people.

Working together to solve a common problem will increase the readiness to create the processes which will address the data governance issues.

I invite your comments and questions.

 

Data Governance Part 3

I would like to return to the definition of data governance given in Part 1 and explore some other aspects of the issue.  The definition was data governance (DG) refers to the overall management of the availability, usability, integrity, and security of the data employed in an enterprise. A sound data governance program includes a governing body or council, a defined set of procedures, and a plan to execute those procedures.

My initial reaction to this definition was that it was some very technical folks creating another complex construct.  However as we experience sharing of data throughout organizations, how does the organization deal with multiple versions of the truth?  The solution is to create one version which everybody shares.   The governance part is making this simple statement a reality.

If we take a case where several people in one division of an organization are developing performance data on many spreadsheets.   The management in this organization realizes that all these spreadsheet require a lot of manual effort and could be done more efficiently by a data warehousing application.  Different parts of this division report sales and revenue in quite different ways.  In addition they do not align with the way other parts of the organization report sales and revenue.  The reason is that each product line has quite different characteristics.  When this data is on different spreadsheets, common definitions are not necessary.

We are asked to help this division develop this application on the warehouse and we recognize immediately the data governance issues.  Currently the governance of the spreadsheets depends on the owners of the spreadsheets.  One temptation is to create a structure that brings the division in line with all the structures of the larger data warehouse.  However for this division, they have no reason to integrate with the warehouse and want to solve their productivity issue. 

We then wonder about the appropriate structure to introduce some management to this process.  Rather than creating a new process we looked at how the division deals with the cross-functional issues currently.  We looked at the budgeting process and how the organization sets priorities.  We found that most of these processes were relatively informal with the leader of the group determining the strategy and allocation of resources.  Each group in the division competes for resources.  We then looked at who was driving the process and who had the most to gain by improved productivity.  Unfortunately the people who were currently creating these spreadsheets and their boss had the most to lose.  The rest of the management group had the most to gain.  We then looked who was supporting the project and their position in the organization.  We then looked at the history of the management team and how they got things done. 

After this assessment, we decided to ask for advice from the head of the division to determine the best way to proceed.  We did not ask simply about data governance, but how to introduce the approach to the organization.  We asked about what the best way to get agreement of definitions of specific things.  We were careful not to use data warehousing terms but business terms. 

We were surprised to find the division head understood the issue and was willing to take leadership on the issue.  She appointed one of her key reports to work with people from the business and our project team to drive common definitions of terms.  This was the beginning of the data governance process for this group.

We will continue to build on the structure created as issues arise.  We will also modify the membership as our experience with the division and their management team increases.  We will eventually develop a set of procedures but we must walk before we run.  We must learn what works and what does not work in a specific organization.   

This case is typical of many people who are beginning to realize the opportunities and challenges of data warehousing.   The lesson is to introduce the disciplines of data governance in response to business issues as opposed to any technical requirement of the data warehouse.