The service-based architecture of ArcGIS Utility Network

Read any article on Esri’s ArcGIS Utility Network and you will find references to its service-based architecture, and the advantages of this architecture over traditional ways of working and interfacing with GIS data in utilities.

But what is a service-based architecture and what does this mean within the context of Utility Network? How will this services paradigm affect the utility organisation and how can we exploit this within the enterprise? In this blog I will answer some of these questions and describe the benefits that the service-based architecture behind the Utility Network brings.

Service-based architecture or service orientated architectures are not new. It is an architectural style of software development where a service defines a discrete unit of functionality and the services are used together in a loosely coupled way which enables them to be upgraded or replaced separately without having to rebuild the entire suite or application. With a services-based approach there are a number of key features which when taken as a whole are intended to make software components more modular and more re-usable.

In the modern web architectures this service-based approach is most commonly exposed as REST (Representational State Transfer) services. When a request is made to the REST service, the resource URI picks up the request, processes it and returns a response payload, commonly in a format such a JSON.

Collections of these REST services are commonly referred to as a REST API (Application Programming Interface), and the API defines how developers should make calls to the services (e.g. defines what input parameters a service request might require) and also what the response will look like.

Esri’s service-based approach

Esri moving towards a service-based architecture is also nothing new and indeed it has gone hand in glove with their transition to a ‘webGIS’ paradigm, with the first services in the REST API appearing way back in version 9.3 (2008). Ever since then, each release of ArcGIS has seen more and more functionality exposed and accessible through the REST API. Indeed, with the release of ArcGIS Pro in 2014 the Esri desktop client became ‘services aware’. This meant that for the first time the desktop-GIS client was interacting with the same services and in the same way as the web-GIS client.

From a software vendor (Esri) perspective this is important, as we are no longer having to maintain completely separate and distinct codebases for different client types. We are beginning to see a convergance of core services and interactions - whether they originate from the desktop or web - consuming the same services. Greater re-use of services increases ‘testability’ and so the services themselves become easier to maintain and the software more robust.

In a disconnected mobile context, our application is perhaps no longer communicating with the same centralised ‘service’. Indeed, the requests we make may now need to go and interrogate and fetch data which is now cached in our mobile device and not on an enterprise or cloud database. However, because we are using the same service contracts and approaches; much of the client application can be blissfully unaware of its context. The application is making the same service requests and getting the same response-syntax and pattern regardless of where or how the request is being handled or where the data being returned by a service is actually coming from. This enables applications to behave and act the same way regardless or whether they are currently connected or operating in a disconnected manner, or whether the data returned is from a server or cached within the device. It is this services-approach which enables Esri to market Utility Network as offering the ‘seamless user-experience across desktop, web and mobile’ and drives the ability of users to access the same functionality whether they are office-based or field-based.

However, Esri’s geometric network pre-dated this services-based approach. It was developed in the monolithic or layered application approach where interactions between clients and application functionality could happen at multiple layers within the application architecture and even as deep-down as direct database interactions. So, while utilities may already have been publishing data and consuming Esri GIS data as services, editing the geometric network features from a web or mobile context and performing network analytic functions such as tracing was often difficult or cumbersome. Some very innovative and successful third-party solutions worked-around some of these barriers, but even where this was achieved it still typically led to a very different, fragmented process and experience for users across desktop, web and mobile. The reliance of a lot of the network functionality and advanced transaction editing employed in utilities being reliant on the ‘old architecture’ were acting as a blocker for wider maturity and adoption of GIS across the utility enterprise.

Services for the Utility Network

With Utility Network the core capabilities of the network have been exposed through services. For example, configured network traces can not only be run through a service, they can also be created and altered. A downstream trace of poles on an overhead electricity feeder circuit can be configured and run in ArcGIS Pro, but then the same trace capability is exposed and available to field crews on their mobile devices.

And it isn’t just constrained to exposing the network services themselves; Utility Network has also precipitated the development of the newer services-based branch versioning mechanisms within ArcGIS Enterprise. This enables all client types to be able to create versions, make edits to the version and then post/reconcile these versions back to the central ‘default’ version. This not only makes editing our actual network records viable via web and mobile, it does so in a way which recognises that in a heavily field-orientated GIS operating environment (e.g. the modern utility) we might have multiple users editing our network both in a real-time combination of ‘connected’; and ‘disconnected’ states. It then allows these edits to be managed, and any conflicts resolved within robust workflow processes.

There are still a few wrinkles though. As sub-centimetre positioning solutions become more prevalent and cheaper for utilities, we are seeing a move towards the absolute sensor-driven positioning of assets. However I frequently see many utilities where the capturing of relative measures of assets to real-world objects (dimensions) is still fundamental to working processes, and in many cases the data held in a higher-level of trust than the absolute position of the asset on the map. Note: The organisation might also artificially sacrifice the ‘correct’ location in order to maintain the relative position, with consequent technical debt building up within the data for a future positional improvement programme.

Whilst dimensions features within Esri can now be exposed via the REST API, and can be read and edited through the desktop ArcGIS Pro client, they are not yet supported within the web Esri Javascript API. This means the capturing and visualisation of these dimensions via web and mobile is not supported within Esri tools and so still requires thought as there are still some functional gaps between clients. Again, I do believe that, over time we will see more and more adoption and penetration of services across all the deeper recesses of the ArcGIS capabilities, and these gaps will get smaller and smaller.

Considerations of the service-based approach for the utility

One of the major considerations for the organisation in regard to this services-based approach will be in the area of integration with other business systems. Typically, in the past a lot of the integrations have been at the database level; perhaps on a batch basis where for example, updates which have happened in the last day are pulled out of GIS and sent to an asset management system or ERP system or vice versa.

In the services-based approach we also need to reconsider how we interface between these systems and our approach and thinking needs to also be at the services level. Again corralling changes and updates in versions via querying versioning services, and pushing interfaces updates in via the services. There are opportunities here too though; interfaces can be made to be more real-time in a services-paradigm; and through careful thought and design we can build interfaces which are more ubiquitous in nature and are less ‘point to point’ between the GIS and other systems. It offers the promise of interfaces which are more loosely coupled, more resilient, more scalable, and more interchangeable.

A stable architecture on which to build added-value tools and products

The services-based architecture also offers a solid base on which Esri partners and 3^rd party software vendors can build additional added-value tools and functionality. The concept of a service contract and service contract versioning allows Esri to change and update their underlying services to be backwards-compatible, with some assurance that existing tools built on these services will still continue to work and function.

For utilities I really think this will be an exciting area. We are already starting to see some added-value, targeted line of business applications come to market which have been built on-top of Esri and the Utility Network and my prediction is that over the next few years the numbers will increase dramatically. This will offer a marketplace for utilities where solutions can be adopted and mixed by the business and tailored to meet their precise needs. However the service-based approach will ensure that technology does not become defunct or adoption of one solution does not impinge or hinder the business elsewhere.

Extending the Esri services

Not only can 3^rd party partners build new applications and tools on top of and through combining Esri services; you can also actually extend the services themselves.

Esri offer a number of mechanisms and patterns for extending their services. These are Server object interceptors (SOI) and Server object extensions (SOE).

• Server object extensions (SOE) allow you to create service operations to extend the base functionality of map or image services. SOEs are appropriate if you have some well-defined business logic to perform that is not easily accomplished using the ArcGIS client APIs. Most SOEs do this by using custom code to work with geospatial data and maps.
• Server object interceptors (SOI) allow you to intercept requests for existing built-in operations of map or image services. This allows you to execute custom logic and alter the behaviour of these services by overriding existing operations in a way that is seamless to existing clients.

Obviously getting to this level of extension is going beyond mere consumption of a service and starts to require a deeper level of understanding of the service-based approach and how services function. It also offers an unimagined ability to offer new, more powerful services for the utility to take advantage of.

I hope I have provided a little more insight into the Esri services-based approach and how this approach benefits the utility and supports and enables the next generation Utility Network technology. I also wanted to explain how utilities themselves will need to understand and adopt their thinking to this service-based model, in order to seize new opportunities.

Any questions, please do not hesitate to get in touch.