How to prevent wild growth in business intelligence and data science tools and at the same time enable innovations in the fields of big data and data science.

In times of digital transformation, big data, data science, machine learning and artificial intelligence, the data scientist has recently become the star among data analysts. The new profession of the all-rounder, who takes the urgently needed view of the data available in abundance and can interpret it, is even treated as the "Sexiest Job of the 21st Century" (Harvard Business Review). With the help of appropriate tools, he can integrate data virtually "on the fly" and in many cases meet the needs of the specialist departments for quickly available data analysis. The ever-increasing performance of the end devices and the sheer number of easy-to-use tools have raised the false expectation in the specialist departments that data analysis is child's play today. Accordingly, the business lacks understanding that the IT department needs significantly more time for similar requests and that the projects are often many times more expensive.

Aimlessness doesn't have to be

Unfortunately, the existing business intelligence (BI) and data warehouse systems (DWH) are often no longer able to serve the steadily growing analytical requirements in a useful time and at low cost, not to mention to integrate those requirements into a clean and sustained cost-effective data infrastructure. It is therefore becoming increasingly common to observe that analytical applications are being shifted towards business while byassing IT processes. It was precisely these island solutions with their sometime infinitely complex Excel documents and access databases that forced companies to get to grips with the basic problem by creating targeted BI initiatives and clean DWH infrastructures: the individual developments and uncontrollably proliferating tools soon nobody in the departments could maintain any longer. Today is no different. Analytical island applications that are difficult to maintain are created without a plan and can often no longer be operated by the department's data guru. If they are not to be completely useless for the company, they are then integrated into a project by IT - tedious and usually with a great deal of effort. However, this back and forth movement is not new. They have existed since IT systems were in existence: the specialist department buys or develops software until the maintenance effort becomes too large or too complex and finally hands the solution over to IT. This leads not only to unnecessary costs, but also to a lot of frustration on both sides. However, such “workaround solutions” cannot simply be forbidden, since one does not want to stand in the way of the company's innovative strength. So how do you get the problem under control and at the same time enable the business to gain important insights quickly and agile from data?

Data Lab, the new Layer

The various analytical applications must first be classified on the governance side in order to bring order to the new data chaos. To this end, I have developed a logical data governance model. It is based on a layer concept with the following components: (see graphic)

• Source layer: This layer represents the data sources themselves. They can be internal and external in nature and can be both structured and unstructured.
• Data Lake Layer, also input layer: raw data copy of the source data
• Governed Masterdata Layer: Here the most important master data of the company are integrated, consolidated and made available in a quality-assured manner.
• Governed Data Repository: This layer integrates the most important transaction data with the Governed Masterdata Layer and encompasses the core area of ​​quality-assured and integrated data.
• Governed Data Mart Layer: A derivation from the Governed Data Repository that represents a new, often aggregated view of the data.
• Data Lab Layer: Here are the silo solutions mentioned above, which are not subject to strict IT governance.

What is striking in the model is the Data Lab Layer, which, in contrast, does not exist in the classic DWH model. It is there to “capture” precisely the analytical island applications of the specialist departments and to allow prototyping and research in the field of data science. All other layers belong exclusively to IT and are professionally developed and operated taking into account clearly defined service levels. These services are an important part of the model. There are assigned different characteristics for each layer - such as the degree of data historization (e.g. source history, mono / bi-temporal), performance, level of data integration and quality, degree of reusability or data security. Other service criteria can include the topicality of the data (real time, every minute, hourly, daily, etc.), the relevance for decision-makers (strategic, tactical, operational), the quality of the documentation or the expected project throughput time. In this way, the input layer receives a much lower service level than the governed data mart layer. At the highest level, it is assumed that, for example, the data quality and performance are excellent, but the data is not available in real time. Services at this level will be more useful to strategic decision-makers. In the input layer, where data integration and quality control have not yet taken place, but the data are available in real time, the data will be of more use to operational decision-makers. Balance sheets that require 100 percent data quality, for example, will be located on the governed data mart layer. In contrast, a clickstream analysis can be set up directly on the input layer, since data quality and integration are not in the foreground, in contrast to the timeliness of the data. At most, it requires an enrichment with customer information, which can be consulted from the governed master data layer. Accordingly, the users of such analyzes are more likely to be found in operative marketing, which, for example, has to make decisions and initiate actions every minute in online marketing campaigns.

Transparency and avoidance of unnecessary costs

By consistently describing the services per layer, hidden services and efforts can be made transparent and service level agreements for analytical applications can be derived. By classifying the existing and future applications in the layer concept, management can always see which analytical applications actually exist in the company, where research is carried out and where there are islands that need to be integrated into IT at a later date. The model also describes the skills that a data engineer or data consumer needs on the various layers. For example, a data consumer who accesses the input layer, must have skills in the area of ​​SQL, Hadoop, Python etc. On the governed data mart layer, on the other hand, it is much easier to consume data. At most, the user only needs to be able to use a simple web GUI or an app. The model also allows controlled degrees of freedom for the specialist departments, so that innovations can be supported using big data and data science. In addition, undesirable developments can be corrected quickly and unnecessarily high costs and redundancies avoided.