Within any enterprise information needed to make decisions is key to the success of on-time, cost effective delivery of services to customers. Due to an all too common problem within many organizations of information (or data) ‘sprawl’ and a lack of data ownership/stewardship most enterprises spend far too much time, effort and money attempting to arrive at a sustainable data strategy. This post gives the enterprise a base data strategy that will be an efficient and cost-effective tool for any organisation undertaking the data journey. Most organizations current data landscape lacks the integral data needed to inform, guide and understand customers, assets and financial positions. Follow this strategy and even you may begin the journey without the costly overhead of finding all the sources of truth in the organizations many data repositories. Further to this as businesses move into the realm of IoT data from wearables, sensors to robotics and block chain will in effect be delayed without a core foundational data capability that can be derived from the base data strategy contained herein.

The base data strategy begins with a Test and Learn approach which will - run and deliver a ‘Pilot’ or ‘Proof of Concept’ that will form your originations ‘Data Lake’ capability, followed with an ingest ‘use case’ of corporate or business data and learn while, assessing the market holistically before, selecting a long-term Data and Analytics approach.

By using this approach, the enterprise is not committing to any technology platform prior to understanding the full capability required. Including the attention to following the operating rhythm within the organisation. A further key driving force to this approach is, ensuring alignment of business and data strategies before selecting any longer-term platform.

Next steps to follow are –

1. Stand up a ‘Data Lake’
2. Ingest data for testing business valuable ‘use cases’
3. Assess the market, and
4. Select the desired approach longer term (e.g. Generate a Business Case and apply recognised project constructs).

Business transformation through data

More and more organizations are transforming the way they do business through data. Identifying more efficient ways of doing business.

Aligning to new business models to service customers efficiently.

Documenting the Business Drivers & Motivations is key to successfully meeting Mission and Vision goals.

External and Internal challenges to operations.

Goals and Objectives of employees and customers to promote a workplace of choice.

While it’s important to “right size” your data strategy, core components are key to the Data organizations success.

Aligning business and data strategies.

Aligning the Data Strategy with Information Security and Classification or associated KPI (Key Performance Indicators) frameworks.

Data Governance and Management/Stewardship to ensure auditability and customer information protection.

Coordinating and integrating disparate data sources for master records.

Management and inventory of source data systems to increase accuracy of reporting.

Business insights that support strategic objectives and outcomes.

Architecture design for sustainability and ingestion of new data types or sources (e.g. IoT, block chain and so on).

Key Areas of Focus

It is paramount that you will need to secure, classify and mater your data at its’ many sources to improve integrity and quality. This will lead to far improved evidence-based decision making. While a lot of businesses consider decision making that comes from instincts, this is not always the best approach especially where you may have sufficient evidence or data that will almost guarantee the best possible outcome/results. To reign in all your disparate data sources and systems, plan out the following regimes;

Data Governance

• Having a Classification framework to aid baseline security controls and measures is essential to complying with Government, Legal and Regulatory Body data management standards, principles, policies and guidelines.
• Implement and promote User Access control through appropriate roles by functions, aligned with responsibilities and maintaining segregation of duties.
• Make an initial pre-population effort of key information into applications/systems with known Data Models and known business validation rules.
• Always keep in the back of your mind that Data/Information/Records must always comply with Public Records Act, Privacy Act, etc. whether data is at rest or in transit.
• Do not forget that Data sovereignty rules are to be obeyed to prevent data loss and loss of control because if litigation results you will be able to enforce your local law jurisdiction by keeping data onshore.

Data Quality Management

• When tackling completeness of data, you must document data attributes in order to enhance the execution of key tasks.
• All the while ensuring you have a strong set of validation rules to apply which will in turn increase quality of data capture.
• Profiling of data and data objects or entities to use to inform concise reporting and accurate application/system usage.
• Data relationships must be documented to ensure predictive over reactive business operations.

Master Data Management

• Single source (Customer, Product, Pricing & Services) of truth
• Any migration of data extracted is transformed to meet standards and loaded post wash of data type to the lake
• Data Assets are a trusted business decision making tool

Data Lakes and Data Marts

• Ensure that all data is treated as a collective rather than independent information assets.
• Stewards and Data Scientists where available should have clearly marked out areas to apply their skills and craft.
• Insights and reports are simple to collate across the enterprise or specific to senior executive area of focus.

Solution Architecture – Data and Application Integration

If your business is like so many others than your current applications and systems integration pattern is most likely to be point-to-point which was easy to build initially but now presents management difficulties and complexities.

Legacy technology integrations typically used very tight coupling of systems which inherently brought complex rigid connections and interfaces. With such interfaces any changes had to be replicated across all connected systems/applications to keep them functioning. This brings about a reactive approach. That means short term fixes and thinking, and it makes switching vendors and products very difficult (and expensive).

Point to point integration meant process duplication as well. Rather than having a central place for business validations, processing logic needed to be duplicated in multiple systems.

Impacts of changes could lead to disastrous outages, as well as maintenance nightmares increasing risk to your business. Trying to promote code into production environments meant trouble in outages and management overheads every time. Often leading to stop gap urgent spending to patch or fix problems with such rigid interfaces with no returns on future value.

A more modern-day approach using Application Program Interfaces (API’s) for integrating systems and applications are business value driven and improve workflow automation.

To deliver true digital services application integration needs to drive efficient workflows (flows of data) in order to deliver more effective customer services.

With a more modern API based approach to connecting data sources you can expect;

Simplified workflow and business outcomes:

• Rules and validation will prevent data ambiguity.
• Improve efficiency to complete services provided to customers.
• Remove duplication of tasks and people reliance (manual intervention).
• Centralized administration of products and services that reduce internal administration and management overhead.
• And the real-time ability to service customers and partners/suppliers.

Consolidation of application capabilities:

• Applications are loosely coupled to enable change in business direction and introduction of new technology without outages or problems.
• User Access through appropriate Authentication and Authorisation to track who does what when maintains strong security.
• Pre-population of key information into applications to increase operational efficiencies.
• And core Data Ownership/Stewardship roles and responsibilities is captured and managed.

New business service offerings:

• Modernized services that enhance bundling and e-commerce (shopping cart) offerings.
• Business diversity that enables new service offerings that can generate alternative revenue streams.

The reason to include Integration as part of the Data Strategy is due to the ever-changing Digital world that requires data to be served at real-time or near real-time rates through modern application designing.

Capability requirements – people and processes

In order to enact your Data Strategy, you must consider people and processes. The desired operating model requires further alignment with business requirements and businesses changing needs.

Decentralized versus Federated operating models will compliment decision making through better data. To ensure you achieve evidence (quality data) based decision making the federated model is recommended for efficiency and flexibility.

Data driven organizations align business and technology in a joint effort to drive effective business outcomes.

Data Engineering

‘Gather’ - focus on the sources of data, classifying the data into different logical domains and consider how the sources produce better quality data by being combined.

‘Manage’ - Each source of data has been ingested, transferred, enriched and stored in central highly efficient data lake. Success will be based on selecting optimal storage formats to ensure efficient usage of the data for further analysis.

Data Science

‘Analyse’ - Given a set of business drivers, data sources can be combined with advanced machine learning and deep learning technologies to answer fundamental business questions beyond the ability of the human brain.

‘Actionable Insights’ - The investments in data management and analysis requires a very specific visualization process that enables key strategic decision makers use of the results of the analysis to make key decisions to grow business value.

Here is the Roadmap

The high-level roadmap for the Business Intelligence (BI) and Data space for any enterprise can be as follows;

1)         Conduct a Pilot or Proof of Concept (PoC) or Trial which sets up foundations:                                     

• Create a data lake to pilot leading solutions.
• Understand data sources, data structures, data quality and challenges to solve.
• Gathers requirements, defines a high-level data model before investing heavily long term.
• Define the target state solution and approach to select vendors.

2)         Post Pilot/PoC/Trial, decide the long-term approach:                                                                                                                       

• Based on Pilot/PoC/Trial outcomes choose path to select vendor and implementation partners. E.g. Agile and/or go to market for an RFT/RFP/Tender/Purchase.
• Identify core data services / stewards (owners) including any new BI tool requirements.
• Present options to senior executives that suit chosen Data management tools.
• Capabilities and operating model changes – people and processes are paramount to successful Data Driven organizations.
• Seek approval of newly created business case to commence building towards target state.

3)         Once long-term approach is agreed iterate to target state:                                                             

• Implement ingestion frameworks and extend pilot data lake to data warehouse model.
• Implement end-to-end pipeline from source to target including reports.
• Integrate additional sources needed to enrich the reporting and data landscape.
• Implement ETL’s (Extract Transform Load)/ELT’s (Extract Load Transform)/iPaaS (Integration Platforms as a Service) to ingest and transform to the data model and frameworks.
• Design further reports that deliver valuable insights for the business.
• Identify where Data Science and/or IoT (Internet of Things) can deliver value.

When you get your Data Strategy right, you can expect data insights to be directly connected to business drivers, answers specific to your organization (unique), clear activities and milestones from actionable insights. And the ability to meet changing business needs with new technologies and integration plays a key role which I will talk about in my next post.

I would love to hear from you and any clarification topics for follow up posts will be most useful. Thank you all kindly for reading my post.