Conway’s Law as an Architectural Tool

Conway’s Law as an Architectural Tool

Conway’s law is named after Melvin Conway. In a work not solely related to IT systems, Conway observed that

organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations

If you are interested in more background and context this is a good article.

The way Conway’s law is often applied to IT systems is that when you examine the systems implemented at an organisation then the boundaries that exist in the organisation are reflected in the software. An organisation that has a clear boundary between Sales and New Product development departments for example, is likely to have separate systems supporting those business units. The implication is that organisation structures have an implicit influence on enterprise architectures. Another interpretation is that an architecture is more likely to be successful if it complements an organisation’s structure as opposed to one that works against it.

Conway’s observations also support why there is often significant IT impact when an organisation restructures or during acquisitions and mergers. A previously successful enterprise architecture can cause friction as the organisation it worked best for is no longer present. As structural changes in modern organisations are common, Conway’s law should act as a reminder to ensure that architectures are adaptable.

Conway’s law is more than an observation – it can be used as an architectural tool in a couple of ways. Firstly, you can assess your architecture or design against the structure of the business it is intended to support. System interfaces should loosely map to organisational and communication boundaries. If a component spans organisational units, you can ask yourself whether it is appropriate. Each business unit may have a different set of often conflicting requirements and priorities and they may change at different rates. Is sharing the same component still a good idea? Secondly you can use Conway’s Law as a guide to structuring your development team. A decomposed architecture could be split into features or components and teams can be aligned around particular features.

However, you look at it Conway’s law implies that the architecture of a distributed system and the organisation structure in which it is deployed are linked. For example, if there is an organisational imperative to deliver a Service Oriented Architecture the topology of the services should be aligned to the organisation’s structure. If you have the perfect service topology which doesn’t fit the organisation structure, then the organisation needs to go through a business transformation. However, it is more likely that your project cannot drive such a large change to the business so your architecture should align around business boundaries. You can use the organisation structure to define boundaries and inter dependencies. You can also use it to determine potential service owners.

You can use Conway’s law to find potential problems in an architecture or to understand why there is a certain degree of friction between your IT and the business. Recently I was involved in a SOA approach where the services were focused around key enterprise data entities.  However, the organization business units tended to use lots of different data entities and in some areas there wasn’t clear ownership for them. The service topology was not aligned with the business. This was demonstrated when business owners didn’t want to take on full ownership of particular Enterprise services – they only wanted to deliver the bits they cared about and not consider any wider requirements. What resulted were services customised to the particular needs of a single business unit, making it not reusable across the organisation. Conway’s law should have told the organisation that if they wanted to have a suite of common services then they needed clearly identified parts of the business to own those services rather than try to find an owner within the existing structure.

So Conway’s Law has a role as an Architectural tool, but does it have other uses?

When organisations are starting on an Agile transformation, they often start with projects. A candidate project is chosen that become an exemplar, a model for how Agile is done in the business. However, many large organisations still have a least one foot firmly in the Waterfall camp, they have PMO’s and service gates and governance, ergh!

So soon the Agile exemplar is asked to fit a fixed scope within a fixed time window. The wider organisation starts to set expectations about what will be delivered when, well before the delivery date. The project has to jump through service gates. What do you mean you have started coding? Where is your Business Requirement Specification? You haven’t had architectural sign-off yet! Service Gate X demands it.

Conway’s law predicts this friction, because the project delivery methodology does not fit the way the organisation is set up. Programme management has to work in a different way, project governance changes as project don’t have a define start middle and end any more. In a highly transformed Agile environment there isn’t a concept of a project any more, isn’t is more likely continuous streams of change.

The key message is that Conway’s law says that you are not going to become an Agile organisation by simply trying to deliver projects using Agile Methodologies. Instead you need an organisational change. The organisation needs to think Agile to be Agile.

Is there such as thing as scope creep in Agile projects?

Is there such as thing as scope creep in Agile projects?

In the old day’s changes to project scope were unloved. They either increased the time to deliver the project or increased costs or more likely both, and that was bad. Change control processes were brought in which were promoted as a positive method to manage the impact of change on a project but instead were promptly re-purposed to inhibit changes to scope. This made project managers happy, the project was delivered to specification, on time and on budget. But the customer was sad as the product they got was no longer what they wanted.

Does this exist in Agile projects?

Changes of scope are good; they are just a question of priority. Changes to scope may involve new requirements discovered through the regular feedback loop baked into Agile frameworks. There may be emerging work discovered when delivering a piece of functionality is more difficult than first thought. Focusing on the immediate work means that requirements will emerge as work further down the backlog is eventually elaborated. In an Agile team capacity is constant. This means the team can deliver a fixed amount of work in a fixed time. As new high priority requirements emerge it simply means that lower priority work will be delivered later or not at all.

Sometimes organisations buy capacity and they have fixed business priorities. New requirement and emerging work set against a backdrop of a fixed capacity still creates that uneasy feeling in some stakeholders when they realise their commitments might not be met and often results in them reaching for the old scope management techniques.

Scrum gives us some approaches to deal with new requirements and emerging work.

When delivering a story, emerging work is unpredicted work that is required to get the story done. This is just the effort required to complete the story, whether it was predicted on not. This gets revealed in the team’s velocity and by definition it is reflected in the work the team forecast that they can achieve in the next sprint. Forecast too little and more stories can be pulled into the sprint. Some teams use tasks to provide visibility of the work they are doing. When work emerges they add new tasks. Adding tasks is absolutely not something that should be avoided. Adding a task to a story in a sprint is not scope creep.

Sometimes the feedback loops build into Agile frameworks uncover new requirements. Ideally a new requirement is a new story. It should not be rolled into the existing story. There are many reasons for this but the one that always comes into my mind is related to the new story’s priority. It would be wrong to assume that the new story has the same priority as the original. It might be low so it can wait until a later sprint. It may be higher so the team need to refocus their efforts on the new priority.

If the PO can’t wait for the next sprint, the work can be added to the current one. Again it is a question of priority. The team’s capacity is fixed. Adding more work means the lower priority work falls out of the sprint and into the product backlog. In Scrum the sprint goal is used as a yard stick to determine whether the story goes into the sprint. If it supports the sprint goal it can go in, if not other options should be considered.

If the nature of the story and its priority means that the sprint goal is no longer valid, the PO has the nuclear option of abandoning the sprint. This needs to be carefully considered. Doing this has an overhead because the team must regroup and re-plan. It can be very upsetting to a highly performing team if their cadence is disrupted like this.  If feedback loops are short, then the time to wait for the team to get to the new work should never be that long, which reduces the chances of needing to abandon the sprint. If feedback loops are longer and requirements are volatile, then the risk is increased.

It is probably worth highlighting an anti-pattern at this point. Agile by its nature embraces change. However constant change is disruptive. I see the current sprint as the calm in this storm. It is the place where the team can focus and get on with the work in hand. However, if the sprint itself is not held in high regard and disruptive change is allowed to creep in, then the fundamental benefits of Agile can be lost. When stories keep changing within a sprint or the sprints are constantly re-planned a churn is created which means the team cannot accomplish anything…, but at least they were being Agile.

Is this what scope creep looks like on an Agile project?

The best user stories are unambiguous. It should not be possible for there to be multiple interpretations of the story, its description or acceptance criteria. However creating an unambiguous story is hard,… very hard. I am yet to see it done consistently across a project. It is the team’s responsibility to create an unambiguous story. Whether this is done as part of sprint planning or in a story kick off is up to the team. When an ambiguous story gets into the team this could cause a problem.

A highly performing team is motivated to complete work. They will have delivered the work based on their interpretation. When the time comes to agreed that the story is done, other stakeholders may have a different interpretation and the team cannot close the story. The story isn’t done, it cannot be released and the team cannot get working on other stories. I have witnessed situations where a small ambiguity in a story is used to lever open the scope, making the story much bigger. The team are then held to account over why it is taking so long to deliver it. The result is a never ending story that is carried forward across many sprints. The team feel that they aren’t accomplishing anything and moral drops through the floor.

Maybe this is what scope creep looks like on an Agile project.

Integration Styles

Integration Styles

If you work in IT and don’t live under a stone, it is likely that you will have come across the term Microservices. It is the new saviour of Enterprise IT. It doesn’t matter what question you ask; the answer currently seems to be Microservices. Out in the Enterprise, SOA and N-Tier architectures are being rebadged as Microservices and the conversations around applying heavyweight governance to Microservices architectures are starting. In the Hype Cycle we are probably near the top of the curve at the Peak of Inflated Expectation and it won’t be long before we are rapidly descending to the Trough of Disillusionment.

I’m not going to use this post to define Microservices nor am I going to try to outline what challenges you might encounter when delivering a Microservices architecture. Instead I’m going to build on some previous posts.

Making Choices, When the Fallacies of Distributed Computing don’t apply and Conservation of Complexity.

These posts cover how it is important to understand the context when making big strategic bets and also how many technically savvy organisations don’t really understand what it means to build a distributed application. They also talk about that reconfiguring your architecture doesn’t remove complexity it simply moves it.

oOo

If there is one thing you need to think about when doing Microservices it is the inherent distributed nature of the system that will be produced. Therefore, you need to accept the fallacies and be comfortable thinking about the styles in which your services will integrate.

There is a reference manual for people working in the field of integration. Enterprise Integration Patterns has been around for some years but the patterns it describes don’t just exist at enterprise level, they are still appropriate at application level when those applications leverage a Microservices architectures.

The first section of the book talks about Integration styles. It discusses the journey from File Transfer, to Shared Database, into Remote Procedure Invocation and then to Messaging. Microservices architectures sit somewhere in between remote procedure invocation and messaging. You might use REST APIs or you might use some type of message queue. There is also the question of whether to use synchronous or asynchronous communication. Messaging would sit on the asynchronous side whilst REST tends to be more synchronous.

integration styles

This transition of styles represents a journey. It is important to understand the benefits that remote procedure invocation gives over a shared database model. It is important to have felt the pain of not being able to make a simple change to a database schema due to the fear of breaking something important. You need to have felt the sinking feeling when you encounter table names such as customer1, customer2 & . And you need to have experienced the frustration in senior stakeholders when they realise that that simple change is going to have to take a big chunk out of their IT budget.

It is the same when considering messaging over remote procedure invocation. When you have experienced the temporal coupling that causes your whole SOA to fail because a single shared service is unavailable you start looking for alternatives. When you see with your own eyes the cascading failures caused when services are available but are taking a long time to respond.

There are two forces at play. Many young developers are unlikely to have experienced of the pain us oldies had in the past. There is a risk that they make the same mistakes again. Secondly many oldies are now in senior positions riding on the success of past projects and have conveniently forgotten about the pain. In hierarchical organisations they are the decision makers and are creating the same systems over and over with different names, N-Tier, SOA and Microservices.

When embarking on a Microservices project look around you. What styles do other projects use, which projects are successful and which are not? What do senior and influential stakeholders understand about modern integration styles and architectures.

If you conclude the rest of the organisational landscape uses a shared database model, you need to ask yourself whether pure message based event driven Microservices architecture is the way to go. Shared database integration is about building a canonical central data model and forcing (ermm, governing) everyone to use it. An event driven approach is very different. It is asynchronous, disconnected and autonomous by design and can take a bit of getting used to. It is possible to deliver a successful project this way, but buy in and experience across the whole team is required. The project team just wanting to go this way is not enough, they must be wide and deep seated experience within the team.

Ask yourself – who is driving this change?  If it is the client maybe there are simpler ways to solve their problems. If it is the project team, are they biting off too much which may cause problems in the future? The reality is that the client may be moving too far from their current maturity level. This increases the risk of failure. It is though you are swinging a pendulum too far and at any moment it could swing back.  Swinging back may mean the project has to change its integration style at a late stage or is left with a hybrid model, some parts of the system using messaging and others going back to old styles such as shared database.

In summary, most organisations are most comfortable with shared database and remote procedure invocation because they are based on mental models that most people can get their heads around.  Whilst some organisations will get asynchronous messaging, care must be taken when introducing the for the first time. Using REST in a Microservices architecture seems a good compromise. It provides a mental model that most people understand however it is possible to layer on more advanced integration styles if the application or business domain requires it.

Assess the context in which the project is being delivered and keep it simple.

Making Choices

Making Choices

When joining a project, it is common that one of the first things you’ll be asked to do is support the team in making choices. Sometimes that might be validating a choice that has already been made and “rubber stamping” it or it might mean you have to make a strategic choice on the client’s behalf. Often in a client / supplier relationship the client wants the supplier to lead this process because the supplier has much wider experience and they can be held to account if things go wrong.

But from a personal perspective you end up drawing up the opinions, assessing each one, making a recommendation and having to stand by that recommendation when people get nervous.

We find ourselves in this position because it is a hard choice. If it wasn’t the client would have made the choice without our input. So how do you define a hard choice:

  • Where there are many alternatives
  • Where there is no clear or obvious option. Each option has a trade off
  • Where once a choice is made it would be costly or time consuming to undo it later

As a technologist the choices you’ll be involved in are likely to centre around architectural styles or technology selection. However, as professionals, we should be able to create a successful system regardless of the technology whether that be .NET or JAVA and it doesn’t matter if it is built from services communicating through REST interfaces or through messaging. There must be something more to it.

It is not just about what you think is the right choice, it is about making a choice that is sensible in the context and environment you are in. You need to get to know your customer and understand how they got to this point. Often understanding what was attempted in the past, and failed, will help you make a better choice now.

An example,

AcmeTech has been a Microsoft shop for years. It’s not because they think Microsoft technology is better than anything else, it’s just that it meets their needs and it is price competitive.

They have started on a large project using their regular development team but have bought in an external consultant to lead the technology side of the project. When that consultant starts they realise that an important project stakeholder is promoting the use of a particular product. It is a product that is well regarded and fits the bill, even the external consultant thinks so, but it requires the development team to integrate to it and configure it using Java. The stakeholder has looked into the license costs and created a business case that concludes that the products costs per feature is extremely low.

The development team say the product is not suitable because they don’t know how to configure it and integrate with it. They would need to rebuild all of their development and test environments to provide an effective delivery pipeline for the project. Instead they propose an alternative product that may fit the bill. This product doesn’t do all of what the Java based product provides, any gaps would have to be filled through bespoke coding, but the development team can integrate with it and configure it through .NET. The existing development and test environments can be used.

The external consultant is asked to choose between

  1. Asking the current team to deliver the project using the Java based product
  2. Replacing the development team with Java Developers and use the Java based product
  3. Retain the development team and recreate a solution with a less feature rich product and write more custom code in the Microsoft stack.

The underlying point here is that AcmeTech believe they will get an inferior solution if they use the existing team to build a Microsoft .NET based solution. They think the resulting solution won’t meet all of their requirements, will take longer to deliver and the project will cost more, than if they implemented the Java solution. They see a Java option solving all of their problems and to be fair it probably would. Their preference is the first option, because the lead time for bringing in a new team would be too long, not to mention the adverse effect it would have on the morale of the existing team.

However, when the external consultant looks at this they recommend the third option.

There justification is that when they look at how the client works they see an organisation that is very effective delivering Microsoft based solutions. Whilst they concede that it might take longer and cost more to deliver the project in this technology the team have a track record of accurately predicting their costs and timelines. On the other hand, the few difficult projects that AcmeTech have had, all have a similar theme: they have attempted to integrate unfamiliar technology. Therefore, the consultant concludes that AcmeTech have no Java delivery capability and therefore no way of predicting the costs or success of the proposed project. Whilst the project could solve many of AcmeTech’s problems they have no capability to deploy and operate that product.

On reading this you may have come to the same conclusion. No organisation in their right mind would choose the Java option in this case. It is obvious isn’t it.

Well yes, in my watered down example it is but in reality things are more complex. In the real world there may be many more options and variations of options. You might not have all the information or you might have conflicting information from different stakeholders. And sometimes the organisation has already made their decision, it is the irrational one that makes no sense and you are the one who has to make it real, dealing with all the flack as you go.

How do you deal with Technical Debt on your project?

How do you deal with Technical Debt on your project?

I assume that most of the people who might stumble upon this post are of a technical background. It is likely that the term Technical Debt is familiar to you and you will recognise one of its many definitions, such as this one by Ward Cunningham and this one from Martin Fowler. As technical professionals we understand that things could have been better, that we would have approached some problems differently if we had had more time and we are well aware of the impact that not addressing problems in your software has on the quality of the project or product.

However, other stakeholders in your project or organisation can struggle with the concept of Technical Debt. The debt metaphor was chosen to aid understanding – not as a complete model that totally describes the problem we are trying to characterise in software development.

In this post I’m going to describe a few “interesting” approaches to handling technical debt on projects I have worked on.

Your debt, your problem

The relationship that software suppliers have with their customers is usually harmonious, but this is not always the case. Sometimes it can be fraught, tense and in some cases, downright hostile. In these situations, the customer will find any opportunity to give the supplier a hard time.

When the conversation turns to technical debt and its impact on a project, timeline, costs and scope, it becomes a stick with which to beat up the supplier.

In the customer’s eyes, the supplier should be perfect, after all they are charging enough! Technical debt is scope creep that shouldn’t have been introduced in the first place and the customer is absolute in their view that they will not pay for it. The customer will not accept any impact on the original costs, scope and timelines that they originally agreed.

So the supplier has to absorb the impact.  This means longer hours and more effort but often there is the question of how profitable the project will be. The team comes under the inevitable pressure from the top. Reductions in quality become unavoidable.

I have experienced a situation where a customer mandated that a supplier does not address technical debt. The customer explicitly removed authorisation that technical debt could be resolved. This happened because the key decision makers at both the customer and supplier did not recognise the impact of  unresolved technical debt or if they did, they didn’t think that impact would matter much. The result was a demoralised team and a product that was of lower quality than it should have been.

Agile in a Waterfall Wrapper

Often when working on an Agile project, if you look closely enough you’ll find it’s a waterfall project in disguise. Are there worried looks when a team doesn’t meet its sprint goal? Are there release planning boards covered in post-its that are arranged like a Gantt chart if you squint a bit? Is technical debt seen as scope creep and something that is bad and avoided at all costs?

You do have to manage scope creep on an Agile project but it should be dealt with as any other prioritisation question during the project. Technical debt will always occur and is something that should be accounted for in your capacity planning. When the process for accepting technical debt into the project takes longer than resolving the problem itself, questions need to be asked.

Agile Activist 

I wrote about Agile Activists here. The Agile Activist is preoccupied with demonstrating that Agile is the “one true way”. Often, to sway people sitting on the fence, they focus on delivering business value regularly above all other things. When the emphasis is on business value, continuous improvement can get neglected because it becomes hard to justify when compared to all the other lovely business features that could be done instead.

This ends with the project team effectively having to ask permission to resolve technical debt. Usually the people who are granting “permission” don’t really understand what is involved resolving technical debt nor the impact of not doing it.

Again a good indication of this happening is the construction of more and more process to “accept” technical debt resolution stories into the project. Do you have to build a case that demonstrates the business value of making an improvement? Do you have to convince a handful of people who are motivated to deliver new business features over making improvements? Do the people you have to convince have any appreciation of what it takes to build quality software?

Them and Us

Some organisations that are in the early stages of their agile transformation, struggle with the idea of marrying architecture governance with agile delivery. They resist architectural involvement in sprint teams and instead create projects with separate architectural teams.

In my experience this creates friction, as the architecture team assumes technical leadership and decision making but they are not delivering any software. They don’t understand what it really takes to deliver a feature and do not understand the trade-offs that the development teams are making every day.

And when the architecture team start managing a technical debt register…

This is for good intentions. They are doing this to ensure that the project aligns with all the necessary corporate standards. This is absolutely required in many industries and if you are the Agile Activist in these organisation, waving the standard of Emerging Design with no oversight, you are very much running the risk of damaging the organisation you are working in.

But back to the satellite architecture led technical debt register. If this includes items such as

  • Solution must use OpenID Connect 1.0 and OAuth 2.0 for API authentication and authorisation and 256 bit keys must be used
  • Solution must integrate into Microsoft System Center
  • UI must be WCAG 2.0 compliant

These are fine, they read like typical non-functional requirements. At best they should be acceptance criteria on stories. At worst they are technical debt items that must be addressed at some point.

What about these?

  • Solution must use logging framework X
  • System components XYZ must conform to this UML model
  • Code must be written like X

If these sorts of things are mandated by a satellite architecture team, then you are asking for trouble. The development team see them as dictates, ignoring them at best or becoming hostile to the architecture team, if you are not so lucky.