Series of Discussion 2: Are your self-service BI tools putting your business at risk?

dilbert-article-nov16

Part 1

Who owns Data Governance?

 

$600M is estimated to be lost annually through poor quality data in US Businesses, according to the Data Warehousing Institute and then there is the fact that poor data is also the cause of many IT project failures – so who should be responsible for the data?

Data accuracy and consistency of data through organisation should be the remit of data governance.  But in reality, data governance straddles both the business needs and the IT function within any organisation and there are no strict rules as to who has the upper hand. Indeed, in some companies it makes more sense to run the project and governance out of IT and in others, it is more logically managed from a business perspective.  However, it is clear that both business strategy and IT logic have a necessary part to play, no matter where the ultimate responsibility lies.

What is adding to the confusion in more recent years, is the dark art of marketing who have introduced the term Self-Service Business Intelligence, giving the impression to the uninitiated that they will be able to slice and dice key corporate data in a free-form manner to be able to extract the necessary insight without the ward of IT.

So, what is Self-Service BI?

It’s all in the name when it comes to marketing, but confusion is increasing in the Business Intelligence sector as there is a propensity to claim to be a Self-service BI Tool and yet in many instances what is on offer is purely a means to extract data.  Self-Service BI tools are “in vogue” and imply an ease of use but many are simply data extraction tools which move a flat file to and from an excel spreadsheet.  However, most users are likely to need information from multiple sources for which you begin to need an understanding of the structure of the databases as well as the skills to do multiple look-ups and data consolidation.

In reality the differentiation between more traditional BI and self-service BI tools is that the new tools are prettier, with dashboards and templates, but invariably they don’t conform to the rules of data governance which should prevent free access to extract and manipulate data. 

Having direct access to the data can be a disaster without the right data governance and ownership rules in place.

Once a level of data manipulation is allowed there is the potential to adjust raw data, pollute it with other data sources and also, when you introduce a personal interaction, you are by default providing room for error. Allowing manipulation allows for errors and miscalculations and this opens up a whole world of pain around data governance

Who runs the Data Governance rules?

IT or Finance – who runs data governance?

Different departments will run different rules but in essence the “Business” needs to take ownership of its own data and the IT function, procedures and processes should support that.  IT need to provide the tools to the business to enable business insight which comes from the intelligent manipulation of data and that is held in an IT system.

These tools and systems need to know the business needs and then IT can put the right processes, governance and solutions in place to adhere to the necessary data governance rules.  These rules can more easily be applied through a data warehouse than a free-form self-service BI tool which jeopardises data integrity. A data warehouse has inherent restrictions and multiple security levels (from standard Microsoft SQL to Active Directory) and so dimensions and hierarchies of the data are restricted to users accordingly.  IT can frame what input is mandatory (even at the field and user level) and what output is allowed and even though users will still want access, if they are only provided the information from a data warehouse they know they can respect the value and accuracy of the data.

IT cannot create the business rules nor should it be held responsible to make business decisions concerning the data. IT can only ensure that electronic rules, based on business rules, operate correctly.   The dynamic over data governance has shifted from the domain of IT to that of the business who may now set policies and processes that manage, maintain and optimize information.  Simply the word “governance” implies a more business-focused approach to managing data, rather than that of the IT department.  And this is where problems can emerge as IT try to lead a project or implement a system where they do not have control over the key elements – the data and the rules that apply to it.  Undergoing such projects frequently uncovers data governance issues, and the familiarity with the data sources and its journey is where the IT team can lend considerable weight to the resolution of these problems.  This is not to say that IT should be overall master of data governance, just that they have an equally important part to play.

A simple 3-point plan is needed;

  • Define the project goals (Business)
  • Define the policies and processes (IT)
  • Define what success looks like and build metrics (both)

Simply put, a partnership between the business team and the technology teams is essential for any data quality management effort to succeed.

Part 2

How to mitigate against self-service polluting data

 

In this, the second part of our discussion , we look at how to mitigate against the risks of self-service BI tools allowing individuals to pollute key corporate data. Having reviewed who owns the data governance, you now need to address the tools in use throughout the organisation.

Points to consider;

  1. The tools that build reports for the most part only connect to the data.
  2. An enabler is required between the front end reporting and the database.
  3. This enablement layer needs to simplify the rules and logic of the data structure and without this layer the user needs database skills as well as an understanding of the reporting requirements of the business… a rare combination. This is why most BI tools available today require significant investment (in time and expertise) before coherent reports can be extracted as systems are understood and data interpreted.
  4. A data warehouse can increase speed to delivery of these reporting requirements as it removes the need to learn complex data schema and can then provide the right information to the right users:
    1. Power users will need ad hoc reports
    2. Most users however will only require a dash board and the ability to drill down to the data but without the ability to extract and manipulate the data
  5. A data warehouse conforms to the data governance requirements of the IT department as it can validate and reconcile true accuracy and then pass that to the BI tools and reporting requirements of the business

The Simple Steps to Data Governance

  1. Perform a full audit on the data base (or databases) and see how data flows through the systems and company, to understand the structure and processes.
  2. Build a data library and from that a data warehouse providing the one single source upon which the business can report.
  3. Implement a repeatable, re-useable, accurate form of input and apply data governance rules via the use of a data warehouse.

Doug Laney, research VP at Gartner stated that Most deployments are ultimately unsuccessful when end users are given access to manipulate their own data — sometimes from unreliable sources.”  This can be corrected with correct data governance which can be included within an enterprise BI tool that traces data source and journey.  Alternatively, any business that has a fully managed data warehouse, resolves this as the data warehouse will have adhered to what Gartner terms “smart data preparation.”

A data warehouse removes the dependency on the BI developers who would have to model, build and test multiple reporting layers to achieve the desired reports. A data warehouse is often used to examine business trends to establish a strategy for the future. The viability of the business decisions is contingent on good data, and good data is contingent on an effective approach to data quality management.

And so it concludes that self-service BI can only be achieved with the correct data governance, project governance and data warehouse underpinning any and all projects.

What defines a Successful BI Project?

Forrester carried out research to find that successful BI project tips (Ref 6);

  • Put the business into business intelligence – you need a business level executive sponsor
  • Be agile and aim to deliver self-service – allow an iterative development
  • Put a solid governance foundation in place – data and project governance

Without proper data governance a company risks a lack of accuracy and accountability.  And yet the pendulum has swung away from the governance of the data by the IT department, towards a preference for Self-Service BI and these do not insist on the correct data governance of control and accuracy.

Stated Wayne Eckerson from Inside Analysis; “Self-service BI is great for users with analytical experience, but bad for users without an analytical background”.

This point was elaborated on by Iain Plunkett of Garrett As soon as you enable end user to makes the decisions about how they access and use data – then you have a huge central problem. You cannot ever then control where that data is being replicated and used. Enter enterprise content management and a central data warehouse – where data is controlled.”

Self-Service BI dramatically increases the need for data governance because of the potential introduction of un-governed data sources and the freedom allowed to business users to create their own data modelling.

The use of a data warehouse can put an effective “data firewall” in place, giving not only one view of the data and thus true accuracy but also by preventing the reintroduction of bad data and only allowing the display of controlled, quality, approved information.

Andrew Mennie

Related articles:

  1. Why Data Governance before BI?
  2. BI and data warehouse – Why wouldn’t you?
  3. BI user adoption – Why not?

References

  1. http://www.datagovernance.com/adg_data_governance_basics/
  2. http://www.itbusinessedge.com/guest-opinions/what-to-do-when-it-owns-data-governance.html
  3. data governance
  4. http://www.computerweekly.com/opinion/Forrester-Best-practice-tips-for-business-intelligence-success
  5. BI implementation
  6. Forester BI project best practices
  7. Self-Service BI vs. Data Governance https://tdwi.org/Articles/2015/03/17/Self-Service-BI-vs-Data-Governance.aspx?Page=1
  8. http://www.jenunderwood.com/2014/02/08/self-service-bi-governance/
  9. http://timoelliott.com/blog/2014/04/qa-self-service-vs-traditional-business-intelligence.html
  10. http://www.kapacity.dk/microsoft-self-service-bi-and-importance-of-governance/?lang=en