The benefits and challenges of building strong data foundations

Data is invaluable but unlocking and leveraging its intrinsic value can be difficult, time consuming and expensive.

In fact, a survey by IBM shows that 90% of data scientists’ time is spent on organising, cleaning and reformatting data.

Many organisations are unable to take full take advantage of their enterprise data, due to the lack of a data management and data governance strategies, an essential requirement for building a strong data foundation.

Strong foundational data can however provide crucial insights into an enormous range of applications, from understanding public behaviour, tracking the spread of disease or making commercial decisions - delivering vital information for critical decision-making.

Indeed, in the United Kingdom, the Government have paid particular attention to the strategic importance of understanding and interpreting data, and they have highlighted the untapped economic potential of under-utilised data through the National Data Strategy.

There are particular challenges associated with establishing a data foundation and understanding these challenges in detail will allow you to overcome them, and see the benefits of effective master data management.

What is a data foundation?

A data foundation is just that: a foundation of data upon which to build. To take this analogy further you can conceptualise that any business applications or resources using this data represent the structures built on top of this solid base.

When building this base, it is imperative that your approach to data management is correct. Geospatial Commission, in collaboration have recently championed a Q-FAIR methodology for data management.

Q-Fair Data

Typically, a FAIR methodology considers the following for data: that it is Findable, Accessible, Interoperable and Reusable. Taking it a stage further, the Geospatial Commission have championed the inclusion of Quality as an important metric by which to rate effective data management, leading to the Q-FAIR acronym.

When considering the wider policies the Q-FAIR data management approach Data foundations are the building blocks for Q-FAIR data systems, and are integral to their implementation.

Data Foundations will include the varied streams of data an organisation has in a truly unified, interoperable, and accessible system. Any business analytics or decision making will depend upon this data foundation, which has become the organisational source of truth.

A definition of data foundations

For the underlying foundation to be viable, there are core tenets it should follow:

• Data must be accurate and complete, particularly in areas pertinent to business decision-making
• A single source of truth - the data must be immutable and the basis upon which decisions should be made.
• Supported with strategy - Your data should be in line with relevant regulations and have a data management strategy to support it
• Interoperable and efficient - Your data must be interoperable across different system in the foundation
• Accessibility is key - Data must be readily accessible, with all siloes removed inside the organisation
• Flexibility and responsiveness - Your data systems and strategy should be flexible, able to respond to changes in inputs and requirements by adopting standardised and future-proof systems.

Free download: 6 Steps to Building Strong Data Foundations

The benefits and advantages of a data foundation

In business, Government and public service fine margins can have significant consequences.

By establishing a robust data foundation, and adopting Q-FAIR principles, you can unlock the true value of all the data in the control of your organisation.

1. Improved data sharing, interoperability and efficiency

Efficiency is a cornerstone of effective decision-making and operational effectiveness. However, embedding efficiency in an organisation without effective data sharing policies, and without a unified data foundation can be a challenge.

By creating a data foundation that features fully interconnected systems that are communicating across your organisation, you can mitigate the need for time-consuming manual transformation, thus driving efficiency and effective analysis.

During the establishment of a standardised data foundation, you will likely find that some data is rarely shared between departments, and any decision-making between these teams could be considered inefficient.

You can also embed the practice of using Unique Identifiers (UID) across your enterprise data. Unique Identifiers make it possible to use and combine data from different sources and across organisational boundaries, eliminating ambiguity about what the data represents. It also enables organisations to reuse existing data while decreasing the need for continual data collection and the number of registers that contain the same data.

Maintaining a holistic approach to your data resources can enable you to make decisions based on the entire lifecycle of the data at hand and understand how it can help make valuable business improvements based on evidence.

2. Unlock value and reuse existing data

Any organisation with multiple teams, divisions and geographically dispersed locations is likely to have huge amounts of data. Before having an effective data strategy and foundation in place it is possible that this information represents an underutilised data resource.

As a data foundation will consist of all data pertinent to you organisation, the implementation process will mean that you need to analyse all of the data in your organisational control, leading to the discovery of this “lost data”.

With more data at your disposal, you can identify more data-based patterns across the organisation, which would be otherwise impossible without carrying out an investigation across all data sets.

Ultimately, establishing a data foundation can highlight your data limitations, allowing you to exploit new information, and unlock the value of all the data at your disposal.

3. Improved ability to respond to regulation

A stringently enforced data foundation and complementary data governance strategy will enable you to adhere to any data regulation standards more readily within your sector, or when tendering for new business opportunities.

We have seen first-hand that data regulation is of growing importance. Many national governments have been taking increased notice of the value of data, whilst also acknowledging the need for standardisation and security concerns. Indeed, the UK Government state as part of their National Data Strategy:

“The government has a further responsibility to ensure that the infrastructure on which data relies is secure, sustainable, and resilient enough to support ongoing digitalisation, economic growth and changes to the way that we live and work… There is also a growing need to ensure that security is incorporated as part of product and system design.” (https://www.gov.uk/government/publications/uk-national-data-strategy/national-data-strategy#data-1-4)

This speaks to wider regulatory requirements seen across government bodies that understand their dual responsibilities to promote innovation and uphold data security.

There are nationally significant projects in the UK that are already using strong data foundations as tools to respond effectively to changing legislation. The National Underground Asset Register (NUAR), in which 1Spatial is helping to build a data foundation, is one such project. Its goal is to collate the underground asset information across the country, ultimately giving us a national view of all underground asset data.

As is to be expected, there is extensive legislation due to the sensitive nature of much of this data. By building a robust data foundation, all unified under a single data strategy, the project can ensure that any data is up to regulatory standards and meets the requirements of the relevant legislation.

4. Improved data analysis and critical decision-making

Business objectives require data, whether that is to measure progress or judge success.

A data foundation will allow you to consider your data as a single entity, rather than disparate, largely unrelated pieces of information.

Typically, when data is hard to reach, inconsistent, unreliable and in siloed locations, aligning data-based decisions with wider business goals is extremely challenging. This creates additional difficulty, particularly when trying to understand how the data can assist with achieving your objectives.

Moreover, data that is consistent, up to date and of a high quality can support improved analysis, forecasting and decision-making.

What are the challenges of building a data foundation?

If achieving a quality-controlled and complete data foundation was simple, then few organisations would be without them, and the effective utilisation of data would be the norm across all industries.

This, unfortunately, is far from the reality.

As with any large undertaking there will be several challenges. These can range in scope and requirement, encompassing decision making and technical implementation difficulties.

1. Stakeholder buy-in

Before you have even started creating your data foundation you will need to gain buy-in from stakeholders. This is especially true for large projects, particularly when you are dealing with large amounts of siloed and disparate data.

Stakeholder buy-in presents a number of problems. Convincing stakeholders of the long-term benefits associated with having data in a state that enables business improvements is vital for securing the required financial support for your project.

2. Lack of data ownership

Secondly, you may encounter stakeholders who are sceptical about the centralisation of data. These may be from departments who are used to maintaining and using their own siloed data. Their resistance may come from the fact that they do not want to lose control of responsibility of the data. The data function in an organisation usually sits in the IT team, and manages data using top-down standards, rules, and controls. Data often has no true “owner” for ensuring it’s updated and ready for use in various ways.

Making teams aware of the benefits to the entire organisation is a key stage in creating your foundation.

Further to this, the potential downtime whilst integrating data into the foundation, or during the transition to using the now integrated foundational data, presents a buy-in question as well as a technical challenge.  Team leads and wider management may be resistant to downtime due to immediate project or customer pressures, and any reservations about downtime will need to be addressed.

3. Scale and siloed data sets

Speaking of organisational structure, large amounts of siloed data is a fundamental challenge that impacts organisations in every sector.

When data is maintained in a variety of formats and in closed-off environments, understanding how to link these systems into your foundation model is a challenge. This is especially critical as the need for validated and correct data is a cornerstone of creating the foundation.

Interoperability refers to the real-time exchange of data between systems and as a data foundation can include multiple systems working in harmony, the power of interoperability is one of the core underlying requirements for the system.

The fundamental challenge is locating and understanding where this siloed data is. Often within large organisations data can be missing or unknown outside of a specific team. In fact, it is not uncommon to find that a large majority of an organisation’s data is underutilised – mostly because it’s difficult to extract and use.

4. Non-standardised data

If you consider that data may be siloed and in disparate locations, it is also likely you will be dealing with information in a huge variety of formats, all at varying levels of quality and completeness.

Even with two sets of well formatted data, the difference in type or technology may prove a challenge when aiming for interoperability.

This can be a challenge when certain data formats, such as LIDAR in the GIS industry, are unstructured by design. Creating a system that can accommodate the nuances of different data formats is a core challenge, and something to be tackled during planning and strategy phases. Identifying and using Unique Identifiers (UIDs) in your data set is important when dealing with disparate data and multiple formats across your organisation. A Unique Identifier is numeric or alphabetic meta-data, completely unique to the feature in question giving data users a way to find the correct data across systems. By knowing there are likely to be differences across your data, but identifying the Unique Identifiers, you can start to connect data, which allow you to link these different formats together.

5. Rubbish in, rubbish out

If data in your organisation is of a poor quality and subsequently bought into your data foundation, any analysis and decisions made will also be of a poor quality. Knowing which data is poorly maintained, how to fix it and how to validate that it is fit for purpose are all challenges you will need to overcome whilst creating a data foundation.

Duplication can cause problems, especially when it is not clear that there’s duplication in the first place. This can happen when your Unique Identifiers do not accurately represent the data or are not identifying that there is any commonality between data (or when there are no Unique Identifiers).

However, it is important to realise that not all duplication is bad, and there are systems in which duplication can be the norm for valid reasons as different systems contain different use cases.

How to get there?

Even with the challenges you encounter whilst creating a foundation and digital transformation strategy, the benefits mean the effort is worthwhile.

The concept of foundational data aligns with the processes of Master Data Management (MDM), and in many respects establishing a foundation is the first step in the journey to embedding MDM in an organisation.

Master Data Management refers to the discipline that connects or combines data from multiple sources in different systems to enable enhanced analysis via business applications, of which foundation data is clearly a key component.

1. Identify

The first stage, which can help tackle the challenges of siloed and unknown data, is the identification of all data sources and who the data consumers are.

By auditing the data currently within your organisation (and discovering the unknown sources), you can start to understand the rest of the requirements, and scope of the work required.

Any data you have identified as pertinent to the operation of the business represents your “master data”, and this is the information you should target to manage.

Once you’ve identified the master data in your organisational control, analysing the metadata from these data sources will give you the power to find common themes, duplications and identify if there are any UIDs (an important part of the process later).

2. Plan, strategise and assign

Once you have gained an understanding of the data across your organisation, it is time to establish the specific strategy you will use to create a data foundation and consider how you will utilise the concepts of MDM further.

An important part of the strategy phase is deciding which regulatory requirements and data standards you will follow. These requirements will influence every stage of data management processes as regulation and standards underpin how you are formatting, and the levels of quality built into data systems.

Alongside strategy, you will need to assign duties to designated “data stewards”, these will be responsible for the maintenance, quality and integration of certain streams of data into your foundational system. By making these appointments, you’ll be ensuring that there is accountability across the organisation.

Through the implementation of an effective data governance strategy that focuses on the quality and completeness of your data, you will more effectively implement and benefit from a fully realised data foundation.

3. Validate and integrate

Having established the master data and UIDs, making sure it is in a fit state for integration is vital.

During the planning stages you will have laid out the relevant regulatory requirements and quality standards your data should adhere to. This is the standard to which you need to validate the data you have.

During integration, the validation and correction of the data is the linchpin of creating the foundation.

4. Govern and manage

Creating a data foundation is not a one off. Maintenance is ongoing and a slip in the quality of integration will undermine the hard work made during the implementation period.

The data strategy you’ve outlined as part of the Master Data Management process needs to be upheld at all times, to maintain the “single source of truth” status your foundation has. This is upheld by your data governance strategy that we outlined in step 1. An ongoing strategy will enable you to monitor, measure, manage and maintain the data quality, integrity and reliability of your system on a continual basis. It consists of a set of policies, processes and people that ensure the management of data throughout its lifecycle. Data Governance facilitates the “three C’s” of data management: Control, Consistency and Compliance.