SDSW23: Data migration - the importance of data readiness

tl;dr

Data migration projects can be complex and daunting, but prioritizing data quality and using intuitive tools can alleviate many challenges.

The Esri Utility Network provides a real-world example of how data migration can be done more efficiently and with greater control, offering benefits such as advanced modeling capabilities and collaborative tools.

The combination of 1Integrate and clear and accurate reporting capabilities has proven successful in reducing time and cost in data readiness and migration projects, as demonstrated by the case of Hunter Water in Australia.

Transcript

Olivia and I are going to talk to you a little bit very briefly about one of our core capabilities, which is data migration. So without further ado.

So data migration projects from the outset can feel very daunting. They can often be large and complex projects. There's lots to consider in terms of the new technology and certainly software changes, potentially hardware changes. They've got the people element you need to consider whether we need new training. What the skills and knowledge that our people are going to need. But at the core of it is the data. That's the key asset.

So naturally at 1Spatial, we take that data quality first approach that Claire spoke to us about this morning. Our tools kind of give us a really nice intuitive toolset to work with and can remove a lot of the pain points from the processes. And we can be agnostic in the sense that we can integrate with various technologies.

So over to Olivia. What we're going to do here is kind of zoom in at a level and talk about data migration through the lens of utilities.

We wanted to basically apply that to a working model or an example where we could prove that a concept we’d built was actually useful and usable. So we chose the Esri Utility Network for this, and it provided really a perfect vehicle for us to test that within a real world context, if you like.

So for those of you who don't know much about the Esri Utility Network, it's a quite a new product. It was launched by Esri in 2018, I think, and it replaces the current geometric network. It's basically offers the users a way of a truly modern utility network, if you like. And it offers many benefits, such as advanced modelling capabilities, edit functionality and collaborative tools. But most importantly to us, it offered other things that meant that the data was at the heart of the product itself. And it integrates things like business logic and rules. So this kind of proved to us that it was a really good example to use, but the product we were designing could be applied to lots of things that need data migration too.

Just in terms of the Esri utility network and putting aside those benefits, it's quite a complex system and it means that users have to learn quite a lot of new concepts and terminology. And in a way that was an opportunity for us to show that it could be done in a more kind of easier way. And again, because the data was a challenge, it was a challenge and an opportunity for us.

And so how did we do this? This is kind of our first plan of our diagram, the migration process itself. And you can see, not surprisingly, we put the data readiness and data at the heart of that product. It's in line with Esri’s own research. They ask their users who are migrating to the new utility network what their prime concern was. And they all said it's the data. So we kind of segmented the process into three parts, if you like. So we start with the data readiness assessment, and this is just an initial phase where we apply 1Integrate rules, 1Spatial software, and these are generic rules. They're just kind of geometric and topological. And at this point in the user has full control over that process. They can run it iteratively as many times as they like. They then move to an intermediary and Esri call this the asset package, this is when they haven't gone fully to the UN model. And again, we have a 1Integrate session, a number of rules here called the validation app. Again the users have full control as with the data readiness assessment, they can run it iteratively.

But the difference here with the validation is that they're running them against a specific Esri Utility Network rules. At this point and when they're ready, again, the customer is in control and then they can migrate to the full utility network model, or they can use a sync tool, which means that they can push that data back to their own model. And this means that the move to the utility network is gradual and it kind of builds on trust and the actual user has full control at every point. So what the kind of the positives of this process, really. So we're running the data quality rules throughout the process.

We're mitigating those customer concerns of data accuracy and completeness, we're removing data issues at the start so we don't have that pain later. We do it in chunks and we also integrate custom rules for customers at the start of the process too. And finally, it's basically the process itself is transparent, and because the user has full control of the process, we build that trust throughout. It's quite a streamlined process. We do integrate different types of outputs are generated for different users. So we have formats like PDFs and GEO databases. And I think, Charlie, you're going to talk about that bit.

Yeah, perfect. Thank you Olivia. So that moves me really nicely on to talking about one of the benefits of our platform, which is our ability to create really clear and accurate reports. And that's really important if we're going to clean our data before we migrate it to a new system. Of course, we need to find those errors in the data before we can fix them. So 1Integrate gives us that flexibility and our ability to create reports in the formats that our stakeholders need. And that is going to be different across the board.

In the example of a GIS analyst, we can provide those in a geo package or file GEO database, which they can simply drag and drop on top of their map. And that really is just, as I say, delivering the results in the format in the way that people want that data presented, making their job much more efficient throughout the process.

So what I want to do now, before we close is just talk about recent customer success, where we've taken them through the process that Olivia's just described. So we work with Hunter Water, who are a utility company in Australia, and we really helped them with that data readiness aspect of their utility network migration. And what was really nice, we deployed the solution as on the cloud, so software as a service and we were ready to go. That really helps reduce the time to value and we were ready to go on day one of the project. And then within the first four weeks of the project, they had completely cleansed their entire source network data. So at that point it was ready to migrate into their new system. Now, that is something they'd been struggling with for some months, actually. And another nice kind of timesaver that we were able to bring into the project is to reduce the iteration cycle using our tooling and the process that Olivia kind of described, from what was previously 3 to 4 weeks to just a couple of days. So huge time savers, therefore huge cost savings.

So just to steal a couple of slides from Claire this morning, this just kind of really shows the results that the customer had throughout those kind of four weeks. So at the start of the project when they did their first data submission, this was the kind of existing state of their source data. Running it through all the rules that Olivia described. I think the red on the kind of screen there represents somewhere around about a thousand kind of data quality issues. And then within four weeks, who doesn't like seeing all of that nice green and those thumbs up. And this was really nice for not only the people in the GIS team kind of delivering that work and fixing the data, but also their kind of senior management and the ability to then report that progress back. And that's what we mean about kind of the data observability across the various stakeholders on that project.

And just to end on some of the quotes from the actual team that we worked with at Hunter Water, the project director was really happy that we're able to accelerate their data migration and get that legacy data quality to 100% the team leader, the project lead, was sort of very happy. He had all the tools he needed to then present that kind of progress upward. And the people, as I say, within the GIS team actually interfacing with the 1Spatial tools, they thought that the reports were great and kind of really helped them be much more efficient in their roles. And as I say, fast track that migration, which had previously kind of taken some time and proven quite challenging.

So I appreciate we've spoken about data migration in the context of utilities, but everything we've talked about there is kind of analogous to any scenario where you're moving data from one system to a new system. And I think on that, we will end.

Back to you John. Thank you. Thank you very much.