Problem to Solution Responses

I've finally had an opportunity to read Julian's response to my article, Problem to Solution.... Because I can't respond on his site, I'll respond here with a promise to expand these thoughts into another article.

First, I generally agree with Julian's thoughts as I understand them.

Julian makes the point that "needs" and "solutions" are tightly coupled pairs that cannot live alone. I think what he is saying is that a real life problem statement always implies some level of solution. If it didn't we'd start every project from a point charge in a vacuum (a phrase I've stolen from my colleague). While this is generally true, I think it's a different vein of argument. What I'm trying to show is that we want to get to the most abstract statement of the problem that meets the business vision. This focuses us on the problem independent of all solutions.

I really bristle when I read terms like "business requirement" and "stakeholder requirement". I used to live by RUP so I understand what they are attempting to capture. My problem is the mindset they put the reader in. I'll post more on this later, but generally a "requirement" is something that "must" happen which implies that someone has already thought about how to do it (the solution), it just has to happen now. My preference is to keep people thinking about the possibilities as long as possible which is a different mindset. To get into this mindset we think about the problem we have, rather than a set of requirements.

Julian does tie the terminology to the perspective, which is something I also eluded to but have yet to fully articulate. One person's "requirement" is another person's "design". I agree with this notion, but I would state it differently. I would say that a solution at one level is a problem at the next level. So we know we want a website for auctions (a solution) what's the nature of that website (new problem). I'm still clarifying this stuff in my head, so the discussion is useful.

Focus the Documentation Effort To Maximize the ROUTE

Documentation is a means to communicate. It takes substantial effort to create good documentation, thus we want to ensure that the effort will provide an equal or greater payback. Just as much effort can be wasted on developing great documentation that turns out to be useless as can be wasted on reverse engineering a system with no documentation. The focus must be to maximize the ROUTE: Ratio Of Usefulness To Effort.

For each potential piece of documentation ask the following questions:

• Who is responsible for creating the documentation
• What is the documentation skill level of people in that role
• Who are we trying to communicate it to
• What information are we trying to communicate
• Where is the information best kept
• What is the easiest possible way to reference the information
• What format is best for the information
• Are there requirements to communicate it in certain format
• When should it be documented
• How many times will it be referenced
• What is the maintenance lifetime of the document
• Is it practical to maintain
• Why should it be documented
• What would be the impact of not documenting it

To minimize wasted effort, use the answers to these questions to find the best documentation form for the information in question. The best form will have the highest ROUTE. The ROUTE compares how much information people get from the document (its usefulness) to the amount of time put into creating the document (the effort).

The usefulness of a document varies based on the perspective. From the perspective of the project, a document is useful if it gives the designers more in-site or information and is used by a large audience. From the perspective of the audience, a document is useful if they get the information they need from it and return to it many times. A useful document may be reviewed once by many people who are then able to provide feedback. Another useful document is used many times by one person to find the information they need quickly and easily. These differing perspectives have differing needs and likely different documentation forms.

Minimizing the document creation effort requires looking at the requirements for format, formality, completeness and the available creation aids. The effort in creating a formal document can be substantially reduced if there are tools available to assist in the generation. For example the use of document templates or document generation from code or models lowers the effort and allows the designer to focus on content. Simplifying the format is, however, the best way to minimize effort. The documentation format is really a communication format. Simple forms such as spreadsheets, sticky notes, e-mail messages, and white boards may provide the same usefulness with far lower effort.

One very effective way to maximize the ROUTE in software projects is to document the solution design and its constraints in the code. The only place anyone will ever look to understand how something actually works is the code itself. Therefore, this is where the documentation should be not only for the low level design (the code implementation) but also for the higher level patterns being used. Unit tests provide additional documentation of the understanding of the problem that was being solved as well as the constraints on the design chosen. Code is documentation.

Every industry and enterprise has its own documentation requirements and software engineers must work within those requirements. In the payment card industry PCI DSS is the big one and has all sorts of documentation and lifecycle requirements to go with it. How to maximize the ROUTE of the documentation in each area is a problem in the domain of enterprise architects to solve.

Problem to Solution - The Continuum Between Requirements and Design

Introduction

Commonly used software development methodologies (waterfall, RUP, agile....) have requirements gathering processes that try to separate the requirements from the design. When captured in isolation from design, the requirements often make assumptions about the nature of the design. And as these requirements become more detailed, they start to look like design. Often they either become unnecessary design constraints or get translated into poorly thought out designs that don't fit within the greater architecture.

The design assumptions that creep into requirements bias the design process from the beginning. Some of these are intentional and meant to push the design in a particular direction or are meant to limit the scope of the effort in order to speed things up. Others are unintentional and simply provide an example of the desired solution. Either way, they only serve to limit the creativity and innovation of the process. Clearly, there need to be constraints on the design and these are valid at the beginning (we cannot start every development project from a point charge in a vacuum); the challenge is removing the unneeded biases and assumptions so that we're free to move toward the best solution to the problem.

Once the problem is understood and we begin to move the design toward the solution, it is natural for the requirements and design to merge. Requirements are constraints on the design of a solution. Therefore requirements are always specific to a solution, and in the end become so specific that they are indistinguishable from design. For example, the design of the flow for placing an item for sale on an auction website requires detailed knowledge of the information architecture. The requirements for the look and feel of the flow are very tightly coupled with the design of the rest of the system.

The Problem

Requirements do not describe a problem to be solved, they specify constraints on the design of a solution. A solution is one answer to the question "how do we do this?", where this refers to the problem to be solved. Normally there is more than one solution to a problem, which means requirements cannot be captured until the nature of the solution has been determined. However, the nature of the solution cannot be known without knowing the problem to be solved. Therefore, identifying the problem to be solved must be the first step when we are faced with a new challenge, be it a new development project or an un-designed aspect of a solution. Next the nature of the solution can be determined, then requirements can be specified so that finally the solution can be designed. Unfortunately, these steps are often mixed up.

Requirements written that imply a problem end up biasing the form of the solution, which in turn kills creativity and innovation by forcing the solution in a particular direction. Once this happens, other possibilities (which may have been better) cannot be investigated. This is a mistake that can be made regardless of the process being used. In waterfall, the requirements document is attempted to be completed before design and therefore must assume a design direction. This can turn into a critical error when the design team wants complete requirements before they do any work on the problem. In RUP, the use cases are often thought through to the point where screen shots and flows are generated before a designer is involved. Again, lack of early designer or architect involvement can force the product owner (or the team specifying the requirements) to set the solution direction. Even in Agile, user stories are often specified in sets or packages that assume knowledge of the direction of the design. While the design team usually gets early involvment in Agile, they must be careful to not assume the nature of the solution based on user stories in a product back log. Biasing the solution can happen in any process and always kills creativity.

This mistake can be avoided by identifying the root problem (or problems) before considering the nature or requirements of the solution. Each root problem or core need can often be expressed in a single statement. Agile explicitly specifies that a "user story" be a single statement. And a "use case" can start with a single statement of the problem. The idea is to unbias the core need from any assumptions the product owner or the design team has about how the need will be met. However, even writing down what is thought to be the core need in a single statement can produce a solution bias. It's natural for people to think about what they want in concrete terms, imagining a form of the solution. The product owner needs a way to convey the problem to the rest of the team and will often use an example of the solution to do so. What is usually needed however is a more abstract statement of the problem. Getting to that statement of the root problem usually requires discussion between the product owner and the design team. Typically this will happen with the product owner supplying the need and the design team removing the biases. The product owner and the design team must work together from the start of the process to identify the root problem.

Describing the root problem in an unbiased, abstract statement is not an easy task. It requires everyone on the team to step back from their own biases of what the solution should look like. Each person on the team must challenge the constraints implied by the statement of the problem they are crafting. They do this by simply asking questions about what is needed. The product owner is usually the best person to answer these questions and defend any constraints, but they too can participate in removing constraints. Project constraints such as time and cost must be completely removed as these are considered when looking at possible solutions. The end game is a set of statements, each with just enough constraints to describe a problem but not enough to unnecessarily bias the solution.

The classic example used in RUP education is that of the online auction. Use cases such as "Browse Auctions", "Sell Item" and "Place Bid" are used to show the process of use case development. These are great examples of unbiased problem statements. No particular solution is implied or required, as side note they also resemble capabilities. But it is unlikely they were originally conceived in this form. They may have started out life as "Browse Items For Sale on Website", "Allow user to create a page to describe item they are selling", or "Send a message to the system to indicate a logged in user's bid". These more closely resemble real life starting points. All imply that the solution is a website. The second implies that the user is creating a page to describe their item and the third assumes that a message must be sent through the system and that the user must be logged in to place a bid. Getting from these initial requirement statements to a set of unbiased problem statements is the challenge.

The assumption that a website interface will be created for the online auction is typical in the software industry. But there are many different ways to fulfill the need of an online auction. For example, the auction functionality could be provided solely through a service to which many different interfaces could connect. User interfaces for this solution could include a website, iPhone app, SMS message. This is type of assumption that must be challenged in order to get to the core need. The core need here is a way for people to buy and sell items in real time without physically interacting.

The reason for having a simple unbiased statement of the problem is to allow the design team to find path to the best solution. The fewer the constraints the more options they have for a solution. One way to arrive at the best solution is to brainstorm. Brainstorming is an art used heavily in other engineering disciplines. It allows the design team to explore possible solution directions, with no critical analysis. Typically a number of ideas are generated which are later evaluated for feasibility and suitability. The best option given the project constraints (time and cost) can then be chosen. What is chosen is simply a path to the solution.

Understanding the real problem allows brainstorming which is key to arriving at solutions that are innovative, the fastest to build or simply the best fit with the architecture. The root problem, will have constraints around it but it will be free of the unnecessary ones.

The Continuum

Given a set of problem statements and form for the solution, the basic requirements of that solution can captured. From this point on the requirements and design can be advanced in sync, with the requirements feeding the design and the design bringing out more questions about the solution and its requirements. This is the beginning of the continuum between requirements and design.

The continuum is a feedback loop where requirements capture and solution design are tightly coupled. As the design advances toward the overall solution, the requirements help to constrain it. The next step in the design must be known before requirements can be specified for it (typically high level design will bring out requirements that can then be applied to detailed design or implementation). The two are inseparable and require the product owner and design team to capture both. While working through a design, questions about the design's look, feel, behaviour, etc. will be raised. The answer to these questions are usually the requirements. But those answers will sometimes imply design that has yet to be done. At these points the design team must step back again and remove the design bias from the requirement, this will often lead to a new problem to be solved. Within the feedback loop of the continuum, knowledge of the problems, requirements, design and solution are continually accumulated.

For example, while designing the solution that allows bids to be placed in an online auction the design team asks how the system knows who the bidder is. The initial answer might be "well, they signed in". However knowing the identity of the bidder is not in the current set of problems to be solved and therefore is not part of anything designed thus far. That any user has signed in is an assumption of one member of the design team. This is not a requirement, it is part of a problem which may simply be "the system needs to know who the user is". Having identified a new problem to be solved the team can now move on to brainstorming the best solution (like locally authenticate bidders or use an OpenID) and then begin to work out its design (where in the flow do they authenticate) and requirements which depend on the solution. This is a contrived example meant to illustrate that solution bias will occur throughout the continuum. And new problem statements can be generated at any time.

A development process can be viewed as a knowledge transfer process. The product owner transfers their knowledge of the problem to a design team. In response, the designer must transfer knowledge of the solution back to the product owner. Each iteration is a cycle of knowledge transfer where the entire team (including the product owner) advances their understanding of the solution and how they got there. The product owner and designers must have a good working relationship because they are all designing the solution, with the help of knowledge transfer.

Understanding how the design of a solution arrived at its current state can be a by-product of effective knowledge transfer. This is the inverse of requirements tracing, the idea that each requirement should be traceable to a specific aspect of the solution design. When requirements and design are viewed as tightly coupled, each design element has constraints on it set by particular requirements. Through the course of the solution creation, each cycle can advance the documentation of what the solution is and how it got there.

Conclusion

Creative and innovative solutions can be created through any software development process as long as the underlying mechanics of how to go from problem to solution are understood.
To be successful the design team must challenge constraints, ask questions about the core needs, brainstorm solutions and recognize that design and requirements are inseparable. Challenging constraints allows the team to identify the problem to solved. Asking questions allows them to understand the problem, the solution, the requirements and the design. Brainstorming allows them to dream up many possibilities for the solution. Finally, the recognition that requirements and design are always tightly coupled allows a design team and product owner to work together toward the best solution.

SOA Principles and Tactics

Below is an article I wrote for the Service Orientation issue of the Microsoft Architecture Journal. The article wasn't published as text but rather as a video presentation which I created and narrated.

XB

Abstract

This article presents six key principles and tactics to successfully transition an existing system into an SOA model. The keys are not a recipe that must be followed in order. Rather, they are a set of principals and tactics that greatly improve the chances of success of an overall SOA strategy. They are evolutionary in nature and an architect must be constantly conscious of them and continually attempting to apply them.

Our systems are constantly in flux because our businesses are constantly changing. This naturally leads to an evolutionary process. An architect's role in this process is to maintain the vision and provide the leadership to achieve it. Maintaining the vision means keeping it up to date with the needs of the business, evolving it in a controlled way. Moving toward the goal requires leading not only the rest of the development team, but (typically) senior management as well. The principals and tactics described in this article assist the architect in leading an organization toward an SOA model, even when that model must change with the business.

Understand the Business Domain and Model

An architect must understand their business's problem domain and the business model well enough to find the natural lines of separation. The lines of separation provide potential boundaries between services. The problem domain will include it's lines of business and it's particular approach to the solution. The pieces that are of most interest are those that can or are being solved with IT.

For example in a business making coats and leashes for dogs and cats, the problem domain includes:

• design of coat and leash products,
• manufacture of product,
• sale of product,
• shipping of product,
• account relationship management,
• material supply,
• ...

Some of these problems have can be partially, if not fully, solved with IT. It really depends on the business model and the solution chosen. The lines of separation will depend upon the business model as well. For example there could be a natural division between dog products and cat products. The suppliers may be the same for both products, but the customers may be different. And certainly the product lines will be different. On the other hand there may be a more natural line of separation between product categories: coats vs. leashes. How these entities are treated and related in the IT end of the business completely depends on how how the company views its business.

Maintain a Dictionary of Domain Terms

It is important that everyone uses the same language when talking about the problem and solution. This seems pretty straightforward however each business and business domain has it's own lingo. The IT end of the business needs to be able to use terms that are not ambiguous or context sensitive. The key to doing this is creating and maintaining a dictionary of the key business and technical terms.

The dictionary of terms will largely come from knowledge of the business domain. The more that is understood about the business domain the more refined the dictionary terms will become. And visa versa; the more that we try to refine the language through a dictionary, the better we will understand the problem domain.

It is also possible to introduce new terms through the creation of a dictionary. However caution must be taken here. New terms must be added slowly to allow uptake and proper understanding by people using them. Typically, the only reason to add new terms would be to remove ambiguity. This can sometimes best be done through a qualifier, rather than a new word. For example, the pet products manufacturer makes harnesses. Different varieties of harness have different webbing and buckles. A particular configuration of webbing and buckle might be called a Carr. Everyone on the manufacturing floor knows what a Carr is and which harnesses include a Carr. Overtime, the specifications of the Carr change until there are 10 different varieties but everyone still refers to them as a Carr. However, when Carr is written up on a pick list the stock guys now have trouble figuring out which one to get. And the supply people aren't sure how much material they need to get any more because there is too much variation between the Carrs. The Carr has become a design template and what is needed are qualifiers to precisely identify which Carr is meant. Is it the Big Fat Carr, or the Skinny Little Carr?

It is equally important to get people in the domain to use the terminology properly. This, generally, happens over time and it starts with the Architect. The principle of Evangelizing the Vision helps with this. It comes down to the Architect using the terms (correctly) whenever he gets the chance.

Maintain a View of the Ideal System

In order to go anywhere, you need a destination in mind. An architect is primarily responsible for leading their team to Nirvana. That is your destination, but what / where is it?

The destination is the architect's view of the best system to serve the business needs. This should be system that is morphed from the current by a reasonable number of steps, rather than being a complete redesign. You should be able to see the system on the horizon. You should also be able to create intermediate views of the system, the steps between where you are now and where you want to be. The intermediate views are based on likely projects that could advance the system toward the goal.

What is a view and what does it consist of? A view is a conceptual, high level design or system model. There can be different view aspects, such as hardware / networking, services / communication protocols, data / access.... The aspects required will depend upon the significant requirements forcing the desired changes. It should contain the architects notes on what each element is (new and old), why it's there, when it might be created or retired.... These aren't formal documents to be reviewed by committee, they are drawings for the architect to keep track of where the system is and where she sees it going.

These views build a roadmap from the current system to the best system. But the best system will be a moving target and thus the map must be constantly updated. New business initiatives, industry requirements and architectural ideas will force changes in the goal and the steps toward that goal. Thus it is important that the views be kept in a format that is maintainable.

A secondary reason for creating and maintaining these views are to communicate the architect's ideas. These views are part of what the architect evangelizes. Thus they must be in the language of the business's dictionary.

Seek Opportunities to Advance the System

For any work to be done some kind of return must be expected. Every organization is different in how it approaches change and updates to its architecture. Most will not drop millions of dollars into a project that simply promises the Nirvana of the most extensible system, and they shouldn't. The return on such an investment, in the absence of immediately enabling new business, is likely to be low.

One very good reason for not attempting a big bang architectural change is the learning process that occurs along the way. No architect, no matter how good they are, can design a perfect system. They will learn new technologies, new information about existing technologies, new patterns, better ways to implement their patterns and so on. Breaking up the change into smaller manageable and methodical steps will greatly improve the chances of success. It also follows two basic tenants of Agile development: always seek to add the most value with the least amount of effort, and delay commitment to the last responsible moment. Make the changes that you need, when you need them, and nothing more.

The key is to find strategic opportunities to move toward the ideal system. Opportunities will come along that can be used to push the system forward. These opportunities will have the return on investment built into them, which makes them ideal for architecture changes that have an otherwise low ROI. They will come in different forms:

• new line of business
• new client(s) or partner(s) with slightly different needs
• integration with 3rd party software
• updates to 3rd party software
• ...

Many of these changes can be opportunities to advance the system towards an SOA model. The trick is to find the right mix of project and architectural change. Any architectural change is going to increase the overall cost of the project. It may also have collateral costs such as changes to work flows, documentation, operations, reports.... These costs need to weighed carefully against the overall project and the potential return in the end. Essentially we're looking for someone else to pay for the changes we want. And it boils down to two approaches:

1. Hide the architectural changes as necessary to support the project,

2. Incorporate the architectural changes with the highest potential return.

Use the first approach sparingly. When it's used too often, it starts too look like the architect wants to gold plate everything. Every project takes too long and costs too much. This approach can be used to kick-start a set of changes, but the road map to higher return must be clear.

The second approach is the method for success. The general theme should be to find projects where the return to the company can be enhanced by making architectural changes (typically adding SOA aspects). The enhanced return will often be the potential to attract future revenue or reduce future costs. The ideal is to find a client who will pay for the changes you need to support them; along the way they pay to better your architecture. But more often it will be an internal business investment to add features, functionality or other improvements.

Each of these opportunities shapes the goal system and the road map toward it. As new projects come along with the potential to advance the architecture, the road map will need to change. The view of the ideal system and the opportunities that might lead there, work co-operatively. Opportunities will come up that enlighten new possibilities in the architecture. Even if the opportunities don't go forward, the architect may adjust the vision of the ideal system based on what she learned from the investigation.

Evangelize the Vision

The vision will not go anywhere, if the architect keeps it to himself. The company needs to see, understand and believe in his vision of the ideal system. The evangelization needs to happen at all levels. The architect needs to get developers, product owners and operations people speaking the same language. That means she needs to use the language from the dictionary she developed. She also needs to continually point people to the dictionary, particularly new employees. Developers need to see where the architect is leading them and why. They can and should ask questions about why he wants a particular set of services rather than some other set (keep an open mind to these ideas, one sure way to kill the advancement is to squash dissenting ideas). Directors and business leaders need to understand the benefits and rewards for moving in this direction rather than some other. They may have heard of SOA and web services but may also be cautious of the cost and complexity.

The architect must find and create opportunities to put forward her vision and the road map to that vision. Those opportunities can be in design meetings, architecture forums and even hallway conversations. The basic premise is to continually use and re-enforce the terminology and vision. When the evangelism is successful, other people in the company will begin using the same terms and promoting the same vision.

Continuously Improve Everything

It should almost go without saying, but everything an architect does should be continually reviewed and improved. This is no different than the Agile mantra of closing a file in better condition than it was when you opened it.

• Continually seek a better understanding of the business and domain
• Continually add and refine terms in the domain dictionary
• Continually update and refine road map to the ideal system
• Continually work on the evangelization pitch.

These principles are not steps or one time actions, they are the main areas of focus. Moving toward an SOA system is best done through an evolutionary process where small improvements are made over time. Each improvement moves the system closer to the goal. It's all about adding value to the business, allowing the business to function more efficiently. An SOA can certainly do that, but only if it is correctly aligned with the business' needs. That alignment will be constantly shifting, so the best way to keep that alignment is through slow steady progress with a focus on continuously improving the alignment.