Problem to Solution - The Continuum Between Requirements and Design

Introduction

Commonly used software development methodologies (waterfall, RUP, agile....) have requirements gathering processes that try to separate the requirements from the design. When captured in isolation from design, the requirements often make assumptions about the nature of the design. And as these requirements become more detailed, they start to look like design. Often they either become unnecessary design constraints or get translated into poorly thought out designs that don't fit within the greater architecture.

The design assumptions that creep into requirements bias the design process from the beginning. Some of these are intentional and meant to push the design in a particular direction or are meant to limit the scope of the effort in order to speed things up. Others are unintentional and simply provide an example of the desired solution. Either way, they only serve to limit the creativity and innovation of the process. Clearly, there need to be constraints on the design and these are valid at the beginning (we cannot start every development project from a point charge in a vacuum); the challenge is removing the unneeded biases and assumptions so that we're free to move toward the best solution to the problem.

Once the problem is understood and we begin to move the design toward the solution, it is natural for the requirements and design to merge. Requirements are constraints on the design of a solution. Therefore requirements are always specific to a solution, and in the end become so specific that they are indistinguishable from design. For example, the design of the flow for placing an item for sale on an auction website requires detailed knowledge of the information architecture. The requirements for the look and feel of the flow are very tightly coupled with the design of the rest of the system.

The Problem

Requirements do not describe a problem to be solved, they specify constraints on the design of a solution. A solution is one answer to the question "how do we do this?", where this refers to the problem to be solved. Normally there is more than one solution to a problem, which means requirements cannot be captured until the nature of the solution has been determined. However, the nature of the solution cannot be known without knowing the problem to be solved. Therefore, identifying the problem to be solved must be the first step when we are faced with a new challenge, be it a new development project or an un-designed aspect of a solution. Next the nature of the solution can be determined, then requirements can be specified so that finally the solution can be designed. Unfortunately, these steps are often mixed up.

Requirements written that imply a problem end up biasing the form of the solution, which in turn kills creativity and innovation by forcing the solution in a particular direction. Once this happens, other possibilities (which may have been better) cannot be investigated. This is a mistake that can be made regardless of the process being used. In waterfall, the requirements document is attempted to be completed before design and therefore must assume a design direction. This can turn into a critical error when the design team wants complete requirements before they do any work on the problem. In RUP, the use cases are often thought through to the point where screen shots and flows are generated before a designer is involved. Again, lack of early designer or architect involvement can force the product owner (or the team specifying the requirements) to set the solution direction. Even in Agile, user stories are often specified in sets or packages that assume knowledge of the direction of the design. While the design team usually gets early involvment in Agile, they must be careful to not assume the nature of the solution based on user stories in a product back log. Biasing the solution can happen in any process and always kills creativity.

This mistake can be avoided by identifying the root problem (or problems) before considering the nature or requirements of the solution. Each root problem or core need can often be expressed in a single statement. Agile explicitly specifies that a "user story" be a single statement. And a "use case" can start with a single statement of the problem. The idea is to unbias the core need from any assumptions the product owner or the design team has about how the need will be met. However, even writing down what is thought to be the core need in a single statement can produce a solution bias. It's natural for people to think about what they want in concrete terms, imagining a form of the solution. The product owner needs a way to convey the problem to the rest of the team and will often use an example of the solution to do so. What is usually needed however is a more abstract statement of the problem. Getting to that statement of the root problem usually requires discussion between the product owner and the design team. Typically this will happen with the product owner supplying the need and the design team removing the biases. The product owner and the design team must work together from the start of the process to identify the root problem.

Describing the root problem in an unbiased, abstract statement is not an easy task. It requires everyone on the team to step back from their own biases of what the solution should look like. Each person on the team must challenge the constraints implied by the statement of the problem they are crafting. They do this by simply asking questions about what is needed. The product owner is usually the best person to answer these questions and defend any constraints, but they too can participate in removing constraints. Project constraints such as time and cost must be completely removed as these are considered when looking at possible solutions. The end game is a set of statements, each with just enough constraints to describe a problem but not enough to unnecessarily bias the solution.

The classic example used in RUP education is that of the online auction. Use cases such as "Browse Auctions", "Sell Item" and "Place Bid" are used to show the process of use case development. These are great examples of unbiased problem statements. No particular solution is implied or required, as side note they also resemble capabilities. But it is unlikely they were originally conceived in this form. They may have started out life as "Browse Items For Sale on Website", "Allow user to create a page to describe item they are selling", or "Send a message to the system to indicate a logged in user's bid". These more closely resemble real life starting points. All imply that the solution is a website. The second implies that the user is creating a page to describe their item and the third assumes that a message must be sent through the system and that the user must be logged in to place a bid. Getting from these initial requirement statements to a set of unbiased problem statements is the challenge.

The assumption that a website interface will be created for the online auction is typical in the software industry. But there are many different ways to fulfill the need of an online auction. For example, the auction functionality could be provided solely through a service to which many different interfaces could connect. User interfaces for this solution could include a website, iPhone app, SMS message. This is type of assumption that must be challenged in order to get to the core need. The core need here is a way for people to buy and sell items in real time without physically interacting.

The reason for having a simple unbiased statement of the problem is to allow the design team to find path to the best solution. The fewer the constraints the more options they have for a solution. One way to arrive at the best solution is to brainstorm. Brainstorming is an art used heavily in other engineering disciplines. It allows the design team to explore possible solution directions, with no critical analysis. Typically a number of ideas are generated which are later evaluated for feasibility and suitability. The best option given the project constraints (time and cost) can then be chosen. What is chosen is simply a path to the solution.

Understanding the real problem allows brainstorming which is key to arriving at solutions that are innovative, the fastest to build or simply the best fit with the architecture. The root problem, will have constraints around it but it will be free of the unnecessary ones.

The Continuum

Given a set of problem statements and form for the solution, the basic requirements of that solution can captured. From this point on the requirements and design can be advanced in sync, with the requirements feeding the design and the design bringing out more questions about the solution and its requirements. This is the beginning of the continuum between requirements and design.

The continuum is a feedback loop where requirements capture and solution design are tightly coupled. As the design advances toward the overall solution, the requirements help to constrain it. The next step in the design must be known before requirements can be specified for it (typically high level design will bring out requirements that can then be applied to detailed design or implementation). The two are inseparable and require the product owner and design team to capture both. While working through a design, questions about the design's look, feel, behaviour, etc. will be raised. The answer to these questions are usually the requirements. But those answers will sometimes imply design that has yet to be done. At these points the design team must step back again and remove the design bias from the requirement, this will often lead to a new problem to be solved. Within the feedback loop of the continuum, knowledge of the problems, requirements, design and solution are continually accumulated.

For example, while designing the solution that allows bids to be placed in an online auction the design team asks how the system knows who the bidder is. The initial answer might be "well, they signed in". However knowing the identity of the bidder is not in the current set of problems to be solved and therefore is not part of anything designed thus far. That any user has signed in is an assumption of one member of the design team. This is not a requirement, it is part of a problem which may simply be "the system needs to know who the user is". Having identified a new problem to be solved the team can now move on to brainstorming the best solution (like locally authenticate bidders or use an OpenID) and then begin to work out its design (where in the flow do they authenticate) and requirements which depend on the solution. This is a contrived example meant to illustrate that solution bias will occur throughout the continuum. And new problem statements can be generated at any time.

A development process can be viewed as a knowledge transfer process. The product owner transfers their knowledge of the problem to a design team. In response, the designer must transfer knowledge of the solution back to the product owner. Each iteration is a cycle of knowledge transfer where the entire team (including the product owner) advances their understanding of the solution and how they got there. The product owner and designers must have a good working relationship because they are all designing the solution, with the help of knowledge transfer.

Understanding how the design of a solution arrived at its current state can be a by-product of effective knowledge transfer. This is the inverse of requirements tracing, the idea that each requirement should be traceable to a specific aspect of the solution design. When requirements and design are viewed as tightly coupled, each design element has constraints on it set by particular requirements. Through the course of the solution creation, each cycle can advance the documentation of what the solution is and how it got there.

Conclusion

Creative and innovative solutions can be created through any software development process as long as the underlying mechanics of how to go from problem to solution are understood.
To be successful the design team must challenge constraints, ask questions about the core needs, brainstorm solutions and recognize that design and requirements are inseparable. Challenging constraints allows the team to identify the problem to solved. Asking questions allows them to understand the problem, the solution, the requirements and the design. Brainstorming allows them to dream up many possibilities for the solution. Finally, the recognition that requirements and design are always tightly coupled allows a design team and product owner to work together toward the best solution.