Reading Data from Disconnected Systems

Reading Data from Disconnected Systems

Reading Data from Disconnected Systems

Imagine a scenario where you are tasked with building a Web Portal for customers to review mortgage offer paperwork. The website needs to integrate with a tried and tested back-end Mortgage Application System. Let’s assume for the sake of argument that this system provides web services to obtain offer paperwork. Whilst the web services were implemented some time ago, yours will be the first project to make proper use of them.

How hard can that be?

A few more requirements,

  1. The Web Portal will be hosted on a cloud provider whilst the mortgage application is hosted in a private data centre.
  2. The web site must be available 24×7 but the due to its age, the mortgage application is only available 8am – 6pm on weekdays and is undergoing maintenance outside these times.
  3. It should be possible to access offer documentation on demand 24×7 from the Web Portal

Still easy? What are your options?

There are two primary patterns

  • Poll the Mortgage Application web services during its operational hours and transfer any “new” offer documentation to the Web Portal
  • Cache data locally and only call the target web services if the data is not in the cache or if it has expired.


This solution is attempting to simulate a publish and subscribe model over HTTP. This is limited due to the nature of the communication protocol. Here the client is acting as the subscriber by listening for events by polling for changes on a web service endpoint.

The success of this approach very much depends on the interface exposed by the publishing system, which is the Mortgage Application System in this case. How easy it is to determine what is new? How often can the web service be called without adversely impacting the performance of the system? After all you don’t want a phone call from the Mortgage Application Systems owner asking you explain why to have cripple it by calling the web services 100 times per second!

Let’s walkthrough an example.

The first thing to realise is that mortgage offer documents must be stored locally at the Web Portal in order to allow them to be accessible when the Mortgage Application System is unavailable, but how do they get there in the first place? Let’s assume that you can query the Mortgage Application System for offer documents by customer ID. In theory it would be possible to call the web service for each customer known about by the Web Portal. The result can be compared with the offer documents stored by the Web Portal and any differences applied to its local data store.

Reading Patterns - Polling

But how does the portal know about new customers? Perhaps as part of a registration process the customer could enter their details and a reference number (Customer ID). However, if you stop to think about it, this is not going to provide an acceptable experience if the customer does this between 6pm and 8am. Do we ask them to come back later? Possibly? This could be dealt with by a friendly error message “your documents are not available at this time” but it is not really in the spirit of requirement 3).

Hold that thought!


In this situation a cache is used to store the result of queries against the Mortgage Application System’s web services. The web services are only directly accessed if the data does not exist in the cache or if it has expired. A retry policy can be applied to these web services calls in order to gracefully recover from temporary connectivity issues.

Reading Patterns - Cached

This solution also forces the question about offer expiry. This will have to be fully understood to correctly implement the cache. You don’t want the customer to review offers that are no longer valid. This is still a concern in the other option but it isn’t as obvious and may be missed.

When the customer first accesses the portal we can retrieve the offer documents from the Mortgage Application System and store them in the cache… except when the customer accesses the portal out of hours.

We have come another route but ended up in the same place.

Comparing the options

The caching solution moves the offer documentation from the Mortgage Application System to the Web Portal on demand. In order to react to the customer’s request, retry logic has been implemented to limit the impact of any transient connectivity problems. The amount of data transferred in one request is predictable. The rate at which data is transferred is not.

The polling solution works by copying offer documents at regular intervals. Here the rate at which data is accessed is predictable (it is dictated by the schedule) the volume of data transferred is not.

Assuming the caching solution is set up correctly there is a high chance that the data accessed by the customer is the freshest it can be. With the polling solution the data can be stale. How stale is defined by the polling schedule.

Often stale data is not a problem. All data is stale as soon as it is accessed. However stale sounds bad to many ears, so sometimes this negative connotation makes it a problem. If you have opted for the polling solution you might be tempted to reduce the polling interval to decrease the risk of the customer receiving stale data.

As the interval gets smaller you are effectively providing the retry behaviour that you have to explicitly build into the caching solution. If the communication link failed this time, it’s OK because it will probably be working again when we try again in five minutes.

Challenging Requirement 3)

Every project has a requirement 3). Many have lots of requirement 3)s. If you have a technology savvy customer you might be able to discuss the flaw in their thinking and get the requirement changed to something more realistic. Often you need to demonstrate the problem. Identify this requirement as a technical risk and use your favourite method to test or experiment. Agile and Lean approaches allow just that. If you are not so lucky, you’ll be working with a demanding client that insists that “requirement 3 is our number one priority” or maybe worse a Project Manager or Tech Lead who proclaims “We can do anything”. Well yes, you can if you have a large pile of cash and a lot of patience.

Is this scenario realistic?

I have taken the disconnected nature of the two systems to extremes in the example but we are working with disconnected systems all the time. As IT professionals we are being asked more and more to integrate bespoke and packaged solutions and work with many different partners and suppliers. Within cloud infrastructures such as Azure and AWS individual nodes disappear surprisingly regularly and transient communication failures are common. Even when you think you have everything under control the Fallacies of Distributed Computing remind you of the assumptions you should not make.

Hopefully this post has highlighted some of the techniques to think about when reading data from disconnected systems.




Conservation of Complexity

Conservation of Complexity

In physics, the law of conservation of energy states that the total energy of an isolated system remains constant—it is said to be conserved over time. Energy can neither be created nor destroyed; rather, it transforms from one form to another.

In IT I postulate that there is another similar law that we all should get familiar with

The conservation of complexity states that the total complexity of an isolated business / IT system remains constant – it is said to be conserved over the lifetime of the solution. Complexity can neither be created nor destroyed; rather, it transforms from one form to another.

If energy is the ability to do work or the ability to move or elicit change then complexity in this context, is the resistance to change in a system. This resistance causes the effort and time to implement a change to increase which in turn increases the cost of sustaining and enhancing a system.

How do I come to that conclusion?

The follow diagrams model the way in which system architecture has developed over the last few decades. Each diagram represents the same set of business problems solved through differing architectural styles. Monolithic systems have given way to client/server or tiered designs which have in turn fallen out of favour to be replaced by service architectures whether the SOA or the Microservices kind.


Each progression above can be classed as simpler than its predecessor. Components and then services have been introduced as a means to compartmentalise business logic in order for it be reused or replaced altogether. The drive to Microservices takes this further. It is the manifestation of the Single Responsibility Principle at architectural level. Building a service that does exactly one thing well, is much easier that trying to weave that code into a monolith. There are no distractions and it is straight forward to articulate the acceptance criteria or what done looks like.

However, one service does not make a solution, so what is the impact to the overall system complexity.


Layering on inter-component or inter-service interaction on to the client/server and Microservices models above highlights that the complexity has shifted to the networks and communication channels between the components that make up the system. It seems that the client and server components are complex to build but simple to connect, whereas Microservices are simpl(er) to build but more complex to interconnect.

Building communication networks between components adds another layer of complexity. Nodes in the network need wiring together and managing. Security becomes more of a concern as the traffic travelling on the network needs protecting. More network hardware is introduced and someone has to manage it. It may be easier to test each component individually but how do you know that the system in its entirety is working? When things go wrong how do you pinpoint the cause?

The complexity has simply been relocated from the software and code into the network and solution management.

People also have roles to play in complexity, after all they are the users of the system. In less sophisticated IT systems, the software plays the part of a glorified filing cabinet. Records are accessed in the system, they are reviewed, changed if necessary and then pushed back. The users and the software are working together to achieve some business activity, often with the users holding relatively complex processes in their heads.

As activities are automated and the burden on people is reduced the complexity moves into the system. New user interface styles are designed and built so users can be more efficient. Workflow systems are introduced that allow humans and systems to communicate more effectively. Monitoring systems are built that enable the proactive detection of issues and provide a means to track and resolve them either systematically or by referring to system operators.

System architecture should not be thought of as something that will simplify a problem. It generally doesn’t work that way. Instead you are choosing an architecture style that arranges the complexity in a form that can be best dealt with. Through system design you are moving not removing complexity.



What does cheap mean?

What does cheap mean?

The project has just started up and the team are discussing the concept for the new solution. Perhaps, they are creating a product metaphor or even sketching out a Product Box. They remind themselves that the customer has cash constraints so whatever the outcome it must be cheap! After all the client did ask for Agile delivery ensuring that they get the highest value features first and that they can stop at any point.

But what did the customer really mean when they asked for cheap?

Did they mean A) cheap to deliver?

Here they want you to provide a walking skeleton as quickly as possible so they can prove the concept. This is quickly made releasable where it can generate value. Only when the solution is proving the concept and generating value is the product enhanced with new features. Only then is a more sustainable technical solution provided and put in place to underpin the product.

The IT supplier is looking to keep the schedule and head count as small as possible. The cost of the resources and the fat trimmed from the delivery plan becomes a primary concern. When issues are discovered during delivery, there is pressure to take the path of least resistance which is often a hack or a workaround or worst, reliance on human intervention to make up for something the software doesn’t do

Or did they mean B) cheap to run

The solution must be reliable and bug free. When a problem is encountered, a clear and consistent monitoring solution must enable simple and timely identification of the cause, allow the system to be corrected and restart any stalled transactions.

The solution should be pay as you go and the client wants to incur no capital costs or other overheads. If the traffic throughput goes down so should the run costs. Obviously, if the product is a roaring success the customer also wants an advantageous cost model that benefits from economies of scale.

A cheap to run system is not necessarily the cheapest to deliver. As a minimum the project team will have to deliver “ops” stories in order for the system to be run and supported effectively. Not everyone has embraced DevOps so perhaps for this customer you’ll have to ramp up a separate support team. Maybe they will mandate that the software has to be integrated with a predefined set of support tools. Maybe they have many other operational requirements too.

Or what about C) cheap in terms of their payroll, e.g., they can reduce head count

The whole point of software is to automate manual processes, right? The customer wants to be able to do more with less. They want to reduce their headcount and therefore the burden on their payroll and they expect that the remaining staff will achieve much more than the original team.

This means the user interface needs to be up to scratch. The user needs to be able to deal with large volumes of data quickly and effectively. When the system needs human intervention it needs to detect the most appropriate user with enough capacity to do the work.

None of this comes for free and the way in which the system will deal with this will require some thought.

Or maybe they meant D) cheap to change and customise the solution (without paying for an expensive IT supplier)

Change is inevitable and you should expect your software to be adapted, extended and repurposed continuously throughout its lifetime. It is unrealistic to work with a customer for a year delivering in an Agile way where the solution emerges over time, shaped by the customer’s priority, only then to assume that nothing else will change for the rest of the solution’s life.

So in this case the customer may be implying that the solution should be configurable so its behaviour can be changed along with the heartbeat of the business. They know that each financial or academic year the business changes and therefore their software needs to change too. The system behaviour should be adjustable by a business user – a development team is not required.

Building a data or configuration driven system is not easy. Not only do you have to develop the system in terms of the business now, but you also have to speculate on what changes may occur in the future. Without care every conversation about the current requirements degenerates into “what if” discussions. It becomes impossible to determine what is required for the present moment, what is required for later and what is simply opinion.

This extra effort in terms of clearing up the speculation and ambiguity should not be under estimated. If you cannot clearly define the requirements of how the system needs to be flexible and package them into unambiguous deliverable units, solution delivery will not be cheap and you are unlikely to meet the objective of having a system that is easy to change as the business changes.

The answer is all of them

Agile projects tend to focus on delivering business value quickly which is sometimes misinterpreted as simply delivering quickly. Delivering value quickly does not just mean JFDI. All the other quality objectives of the solution must be understood and there must be a means to deliver them too.

This is where architects add value on Agile projects. Architects have experience of looking at the bigger picture and focus on the non-functional characteristics of the solution. They can work with the Product Owner and the development team to help them understand how the solution will fit within the organisation. When Architects are asked for a cheap solution they can help the customer understand which kind of “cheap” they mean.




How Podcasts saved my (professional) life

How Podcasts saved my (professional) life

Update: Here is a large list of developer and technology focused podcasts.

Apologies for the clickbait title. It’s not that extreme, but Podcasts have had a huge impact in my professional life over the last year. To understand how, we need to go back in time to late 2014.

At the time I had just finished a Lead Architect role, which was very hands off as you might imagine. I found the whole experience frustrating. I would see problems coming, but there was very little I could do about them. When they occurred, as predicted, I tended to be the focus for lots of pressure and I was the person that had to pick up the pieces.

My next role was a complete contrast. I was thrown into a project in its early stages that needed prototypes doing for various technologies. I had a passing understanding of some but no knowledge of others. After spending far too long out of the code my Impostor Syndrome was working overtime. I was slow to pick up the new technology, I was applying outdated concepts and I felt out of my depth having detailed conversations with other members of the team. The project demanded long hours initially, so there was little time for the background reading that would normally fill gaps in my knowledge.

I needed a way to rapidly get up to speed that fitted in with my daily work routine.

My commute to this client was 1 hour each way if I was lucky, and up to 2hrs if I wasn’t. I was already listening to comedy podcasts to manage the mind numbing boredom of commuting on Britain’s motorways and A roads at rush hour. Podcasts such as The Bugle, Frank Skinner & Richard Herring’s Leicester Square Theatre Podcast (RHLSTP, I’m a cool kid) were an excellent way to use my time commuting.

It occurred to me that if I added some technology focused podcasts to my rota it might help with the challenges I was having at work. After some initial searching I found SE Radio, DotNetRocks, HanselMinutes and some that were being put out by Thoughtworks. I was a little sceptical because looking at the show titles there didn’t seem to be much in common with the work I was doing.

At first I might not have understood all the details straight away but I found the content rubbing off on me. At work, when looking at some of the newer stuff I had a better appreciation of the IT landscape I was operating in. I was starting to better recognise the problems that certain technology was created to solve which gave me better understanding of how to apply it in my particular circumstances.

Over time conversations with colleagues changed. More and more I could support my points with context not just from my previous experience but also from things I had heard whilst driving to and from work. In a few short months I had plugged many the gaps that a year of working in a hands off role had created in my knowledge. As further benefit I could share my experiences as we ramped up the team later in the project. I created a hand-picked “listening” list for developers that covered some key concepts related to the project.

After a year it feels like I am regularly dipping into what is happening in the industry. Sometimes the show title might put me off but then, when I listen to it I might find a hidden gem, or sometime later the subject becomes relevant at work I have an advantage because I already have a passing understanding of it.

When I reflect on this I realise how important it is for technologists to have broad horizons. Often we get stuck working on the same project or same technology stack for several years. You will often hear people saying they have 3 years’ experience of this or 5 years’ experience of that. When you look closer you’ll find that some of these people have been repeating the same 6 months experience several times. I believe the wider horizon I have gained over the last year makes me more credible and makes me better at applying the appropriate solution to a problem and not just the one that I have the most experience in.

Podcasts didn’t save my life – they made me look up and look forwards in my professional life. They gave me renewed enthusiasm for technology and made my commute a little more bearable.