3 steps to optimising your Data Supply Chain

Common challenges in the data supply chain

1. Different data goals

Let’s start from the beginning – when harvesting raw data in the field. Data is typically collected with a specific use in mind, and even for a specific type of user or persona. For example a surveyor might digitise a new housing estate with the goal of recording building footprints. They might not be considering building access points, address data or other metadata relating to the use of the buildings. Perhaps it is a multiple occupancy building? Or it might be a commercial building such as a corner shop. These details may not be accurately recorded as they are out of scope of the data collection programme, but they might present huge value to other data use cases (for instance for the purposes of tax collection in multiple occupancy buildings).

2. Varying techniques and formats

When data is being collected by different parties, there will often be subtle differences in format or schema. This could be down to tooling, capture techniques, varying levels of quality control with each supplier organisation, or simply the habits of a particular data collector.

Conforming to given requirements is crucial at the data capture stage. There are often standards and specifications that data must adhere to, ideally during capture and before data submission, but it is not un-common for the data to be checked after creation, and upon receipt. The implications and risks of not conforming to these standards at all are delays, re-surveys, inefficiencies, and ultimately - increased costs.

3. Volumes of data

But the challenges don’t stop at this end of the supply chain – once collected, the data needs to be processed. The volumes of data being collected grow exponentially every day, and this can be a monumental task for data aggregators and processors to address.

Think of organisations such as the US Census Bureau, who every 10 years must collect data from all US citizens, combining boundaries and road data from state, local and tribal governments alongside addressing data from the postal service. Similarly, the UK’s National Underground Asset project (NUAR) has around 650 asset owners (users), who each might have around a dozen groups of data. Some of those groups run from 100s to 100,000s of features.

4. Data in flux

Besides collecting new data, another challenge is changes to existing assets, especially if a third party incorporates that data asset into its business process. This is where the feature lifecycle and unique identifiers are key. For example, if a field or pipe is split in two, the ID of the original feature may stay with the larger feature, and a new ID would be created for the smaller. What other systems or data use those identifiers? Do other systems need to be informed of the change? It is crucial to have business processes that perform appropriate matching with other business data to deliver effective master data management.

If these large quantities of data are not processed quickly enough, consumers will find the data is already out of date or “stale”, and all that hard work will have been wasted. Likewise, if the data is not stored in a structured and useful way, how can data consumers find what they need to know, make informed decisions, or even know what is or isn’t available to them?

5. Interoperability

Interoperability, or the ability to exchange information securely between systems, is crucial to the access and delivery of your data throughout the supply chain. This can be extra challenging to ensure across multiple systems, especially where legacy technology may be involved. Any process you design for your supply chain must be adaptable and scalable for ever increasing data loads and security requirements.

The advantages of a strong data supply chain

Beyond simply avoiding the problems we’ve outlined, building a strong supply chain for data also offers new opportunities, especially if we’re considering data as a product in its own right (which will deliver continual value for the organisation).

1. Re-use and reduce

Maximising data sharing and re-use allows for a greater variety of end-products, accelerating innovation and may help you to address new markets. It can also reduce the amount of work having to be carried out in the field – no need to unknowingly collect data twice if your processes and pipeline are transparent enough. Ensuring your data is correct and complete at the time of collection means fewer site visits, and saves time, money and frustration!

2. Integrate and innovate

Location can often be the key to tying together different data sets, as everything happens somewhere.

Improving your data discoverability with good structure and indexing can be the key to innovation. After all, you need to know what ingredients you’ve got before you can come up with new recipes. What if the key seasoning ingredient was hidden at the back of your cupboard and you never knew it was there?

The collation of all assets in one format and hierarchy, enables you to identify real-world assets in a structured way that all stakeholders can locate and understand.

3. True and trusted

Establishing a dependable data foundation is a mark of success. Enforcing data quality standards can help you easily demonstrate compliance when required, and establish that your data and products can be trusted. The closer your data is to “live”, the more valuable the intelligence that can be leveraged from it.

It is also important that you don’t default to accepting the bare minimum just to get asset data incorporated into your data management platform. Strong processes, such as automated rules, mean you can enforce that the data submitted to you is of a sufficient standard.