Working in a multi-team, multi-program, multi-product environment brings several challenges. One of those is providing a smooth flow of work to teams, and incorporating their feedback, while staying responsive to the needs of the business in a changing environment. Managing the portfolio backlog is a critical piece of the solution. This Experience Report documents several years’ experience working in such environments. The focus of this Experience Report is specifically on managing the portfolio backlog, not the full scope of what could be considered under a portfolio management strategy and implementation. We have found that getting the portfolio backlog management strategy right is a key element in the success of the overall portfolio management approach.
1. INTRODUCTION
It sounds like it should be straightforward: create a list of all the work the organization has to do, get people together, prioritize the work, make changes as needed. The reality of the solution proves to be far from obvious. The attempt to define a portfolio backlog approach brings the organization’s agility into sharp focus. Those business units that make it work effectively find that having an approach for managing the portfolio of work contributes to overall organization responsiveness. If we are to take an appropriate approach to solving a problem, then understanding the nature of the challenge is critical. Many issues we face in managing large portfolios are complicated; others are complex. These require different responses. Using the Cynefin framework as a lens, this paper shows examples of both complicated and complex challenges in portfolio management. This Experience Report will share several lessons learned, and highlight some pitfalls to avoid, when designing your own portfolio backlog management approach. Section 2 provides some background context, including a brief description of A3 problem solving, Cynefin framework, and Human System Dynamics, and why they are relevant in the context of this experience report. The majority of the paper describes specific challenges, including how they were resolved and lessons learned. Section 9 summarizes some lessons learned, which includes some of the advantages we have seen, some of the challenges that still remain, and some recommendations based on experience. There is a references section at the end of the paper with details of relevant papers, books and links that are referenced throughout. These are good sources for further information.
2. Background
I work with large organizations that are located in multiple countries, have hundreds to thousands of employees, and sell products, solutions and services to a wide variety of customers and markets. The challenges described in this paper, and lessons learned, are aggregated from multiple businesses over several years. Teams and organizations are complex adaptive human systems. To deal with problems in portfolio management, and other problems, it helps to have some frameworks to help us make decisions. Two frameworks I use regularly in this context are the Cynefin framework and the Containers, Differences and Exchanges (CDE) model from Human Systems Dynamics (HSD), both described later in this section. I use A3 problem solving approaches [1]. To reflect the learning, the problem descriptions in later sections reflect most of the sections of an A3 problem solving report, including context (or background), problem description, impact, countermeasures, and actions.
2.1 Portfolio management
The organizations I work with need to balance investment decisions across multiple business areas, market segments, groups and teams. There are often portfolios within portfolios, meaning we are dealing with multiple levels of products, solutions, programs, and projects. For simplicity, think of a solution as comprising multiple products, and a program as comprising multiple projects. The programs are generally run with teams in multiple geographic locations, and the program teams need to work with customers around the world.
2.2 The Cynefin framework
The Cynefin framework is a domain model designed to help decision makers understand their context so they can take action appropriate to the domain they find themselves in [2]. The domains of the Cynefin model are obvious, complicated, complex, chaotic and disorder. Different problems we encounter in portfolio management fall into each of these domains. The examples in this paper note which of these domains the problem is mapped to, with some explanation for the choice.
Complicated problems are the domain of good practices and subject matter expertise. In Cynefin terms, the appropriate response is to sense, analyze and respond. To tackle these problems we use a combination of research, analysis, consultation with experts, and managing and enforcing processes. Complex domains are the place of emergent practice. In Cynefin terms, the appropriate response is to probe, sense, and respond. This is, literally, where the practices for how we deal with a problem need to emerge through experimentation (or probing the system). To tackle these problems we design and run small, safe-to-fail experiments.
The reason we map challenges to the appropriate domain is so that we can take the appropriate response when trying to solve that problem. If, for example, we treat a complicated or complex problem as if it was obvious, and we try to apply best practices, we’re not going to make any progress in solving the problem. Even worse, there is a risk of falling into chaos because we are not really addressing the problem.
2.3 Containers, Differences and Exchanges (CDE)
Human Systems Dynamics (HSD) provides a useful lens through which to understand self-organization in teams and organizations [3]. Self-organization is a key property of complex adaptive systems and agile [4]. Three factors shape patterns and form the conditions for self-organization in human systems: containers, significant differences, and transforming exchanges [5]. These three elements form the CDE (Containers, Differences and Exchanges) model in Human Systems Dynamics. A container sets the boundary for self-organizing systems [3]. The container’s purpose is “to hold the system together, so relationships between and among agents can be established” [3]. In our context, a development team is an example of a container. The portfolio itself is an example of a container. Any human system can contain multiple containers simultaneously, and the agents in the system can be part of multiple containers simultaneously [3]. For example, an engineer is part of a development team, a program team, a business unit, an organization, and a company – all at the same time. A difference is something that represents the potential for change in a container, and is a necessary condition for self-organization to occur [3]. Differences determine the patterns that emerge in self-organizing systems. Examples of significant differences include power, levels of expertise, skills, quality, cost, gender, race and educational background [5]. We want to have a significant difference of perspectives in the portfolio team, e.g., product management, engineering, architecture. Considering the portfolio as a container, want a significant amount of difference between the items in the portfolio. Examples include diversification of products and solution. For a given product or solution, the portfolio backlog should have a significant mix of new feature, architecture work, fixing defects, addressing technical debt, or researching new ideas. A transforming exchange is the interdependence between agents in a complex adaptive system, and is critical to the ability of the agents to self-organize [3]. Examples of transforming exchanges include synchronization meetings that keep the work of two or more teams coordinated, and the flow of work from the portfolio team to program teams to development teams.
2.4 Problem solving approaches
Cognitive Edge, the company behind the Cynefin framework, provides some templates for working with problems in each of the domains [6, 7]. The templates are useful for providing a structure that steer us through the problem-solving thought process, in a way that is appropriate to the domain. For example, the template for managing complex problems asks us to articulate the proposed actions that will move us forward. It gets us to think about the expected signs of success, and how we might amplify those. It gets us to consider possible signs of failure, so we can be prepared to dampen those. The template for managing complicated problems, on the other hand, guides us through the process of researching multiple, potentially-relevant options, consulting experts, making a choice, and then justifying that choice. We use Cynefin action forms, and A3 problem solving reports, to address complex and complicated challenges. Sections 3 through 8 describe specific challenges we addressed using these techniques. The format in which this report presents the challenges reflects the problem solving approach.
2.5 Why discuss theory in an Experience Report?
This is an experience report. My experience is that understanding the theory helps inform our practice. As practitioners, we need to understand the fundamental principles of what we’re doing. If we don’t understand why something works, we might get lucky, and things might work out fine for a while, but we won’t really understand why. Furthermore, when we run into problems there’s a risk we won’t fully appreciate the context or understand the causes. This means we would miss out on proven approaches for tackling challenges in complex domains. If we want to scale our practice, we need to understand why that practice is working at a smaller scale. When it comes to managing larger portfolios, it helps to first understand the patterns and principles at play in the dynamics of the organization. The specific problems are expanded upon in Section 3.
3. Challenges in Portfolio Backlog Management
Table 1 is a brief summary of some of the core challenges we encounter in managing portfolio backlogs. The table describes the nature of the problem, and also notes the Cynefin domain that we place the problem in so that we can begin to solve it. Note that the term initial is important here; we want to make the problem easier to deal with, by potentially moving it to a new domain. Each of these challenges will be addressed in more detail in sections 4 to 8.
Table 1. Summary of challenges in portfolio management
4. Choosing a Focus when Structuring the Portfolio
4.1 Context
Larger organizations have many groups and teams. These might be organized into business units. The organization is typically responsible for a set of products and/or services. The products and services might be customer-facing or used by internal consumers in other groups. The organization targets multiple markets, and develops products for mass-market sale, rather than for just a single customer.
4.2 Problem and impact
Any decision around structure has consequences. Many organizations choose a default structure based on the organization hierarchy. This makes the portfolio structure susceptible to organization changes, e.g., if the management structure changes it has an immediate impact on the structure of the portfolio.
4.3 Cynefin domain
This is a complicated problem. There are several options to choose from, and a combination of research and consultation with subject matter experts helped understand the problem. We looked at case studies of what other organizations were doing (not all from an “agile” perspective), and read widely on the subject. We experimented with multiple options in different parts of the business, and incorporated feedback and lessons learned.
4.4 Countermeasures
Use a Value Stream perspective to understand the end-to-end flow of how the organization delivers value to its customers. The ‘Stream” can be more of a chain or network for larger organizations. Use this to get a clear definition of the products and services provided to customers.
4.5 Actions taken
Organize the portfolio around the products and services it provides. Factor out common components that are used in multiple products. For organizations that develop solutions that consist of many products, organize by solution. This leads to a layering structure. Examples we use in different portfolios include the following:
- Organize by product: E.g., Product A, Product B, Product C.
- Organize by operating system: E.g., iOS, Android, Web, Windows, Mac, Linux, etc.
- Organize by platform: E.g., mobile, Web, desktop, Cloud
4.6 Intervention type
In CDE terms, this is primarily a Container intervention. It involves changing how we view the organization. This has the further effect of influencing the Exchanges with other Containers, e.g., customers, other organizations that depend on the provided products, components and services, and other stakeholders.
4.7 Resulting benefits
The organization is more resilient to internal organization changes. Reorganizations are less likely to impact the operation of the portfolio.
4.8 Signals of success or failure
Look out for churn in the portfolio structure. When new products or programs are added, do they fit naturally in the defined structure? Check in periodically to make sure the structure is still coherent with the organization strategy.
5. Defining the Baclog Content
5.1 Context
We need a backlog of content for organization portfolio. Depending on the size of the organization, there might be multiple layers of backlogs that are applicable.
5.2 Problem and impact
Many organizations start off assuming it is the sole responsibility of the product managers to define the backlog. Other organizations don’t include product management, and instead define the portfolio content from an engineering perspective. Many organizations neglect to include architects. The net result is that the definition of the backlog content is not representative of the core perspectives needed at an organization level.
5.3 Cynefin domain
This is a complicated problem. There are several options to choose from, and a combination of research, analysis and consultation with subject matter experts helped understand the problem.
5.4 Countermeasures
Make sure the portfolio backlog items are properly defined. The portfolio backlog is not the place for user stories – that’s usually the wrong level of granularity. One challenge with large programs and products is that user stories can become very granular. If we aim for fine-grained user stories that can be completed in a few days, then there can end up being hundreds, or even thousands, of user stories. We need a higher level of abstraction to make sense of the work that needs to be done. We typically use features as the right level of granularity, where a feature is then decomposed into many user stories. When defining these features, we need the input of multiple perspectives.
5.5 Actions taken
Distinguish between marketable, customer-facing features and architectural features. Architects are typically responsible for defining the content of the architectural features. Product managers are typically responsible for the definition of the business features.
5.6 Intervention type
This is primarily a Difference intervention. We want to ensure the backlog has a healthy mix of different types of portfolio items.
5.7 Resulting benefits
Roadmaps and planning are more accurate because they consider a wider set of elements than simply features.
5.8 Signals of success or failure
Success signs include an ongoing, healthy mix of backlog items. Potential failure signs include the exclusion of one or more types, e.g., missing architecture features.
6. Enabling Flow Through the Organization
6.1 Context
We have many hundreds of teams using an agile approach to getting their work done. These teams depend on a steady flow of information to get their work done. Teams have backlogs that contain user stories and other work.
6.2 Problem and impact
There are multiple streams of flow in large organizations. These streams are often organized into programs. Each program has multiple teams. At a program level, we need to provide a steady flow of work to each of the teams, so there are meaningful user stories available when the teams are ready to pull new work. If there is not a steady availability of well-formed user stories, this introduces avoidable delays.
6.3 Cynefin domain
This is a complex problem. Different organizations need to learn how value flows for them; they can’t just copy & paste learning from other groups. This learning is achieved through many small experiments that probe the system.
6.4 Countermeasures
Have the portfolio team meet regularly to define features. Have a regular cadence of meetings with product owners to create user stories based on the features. Define a “Definition of Ready” for features and another for user stories. Use Definition of Ready as a focus for enabling the steady flow of work [REF XP paper].
6.5 Actions taken
Define groups of program managers, engineering managers, architects, and product owners that work together on related areas. Their core responsibility in this context is to work with the teams to get features and stories to meet their Definition of Ready [8]. This means defining containers with a deliberate attention to the different skills and perspectives within the container, and the exchanges that this makes possible with other containers (development teams, portfolio teams).
6.6 Intervention type
This is primarily an Exchange intervention. When starting out, we don’t necessarily want to change the containers or the differences within those containers; we want to understand things as they are. Once we have a good visualization and understanding of the flow, the first step is usually to start improving the communications between different containers, with the goal of reducing hand overs and minimizing delays.
6.7 Resulting benefits
Having a definition of ready encapsulates all the things that need to be considered when defining features and stories.
Having the right group of people formally come together means that the right perspectives are taken into account when getting features and user stories ready.
6.8 Signals of success or failure
Possible failure signs include ignoring or relaxing Definition of Ready, or falling back to a state where development teams are delayed because of upstream dependencies.
7. Portfolio Backlog Management Meeting
7.1 Context
We need a forum for managing the portfolio backlog. Recognizing how to structure the portfolio backlog, the importance of flowing work to the teams, and the need for cross-functional efforts in defining the backlog content,. There are similar principles at play as when managing a team or program backlog, but the scope of the portfolio backlog introduces additional considerations.
7.2 Problems and impact
We need a cadence of meetings to keep momentum going. However, its not always obvious what the cadence should be, nor who the right mix of participants are.
7.3 Cynefin domain
This is a complicated problem. There are multiple possible options, many of which could work in a given situation. The response here generally includes investigating several options, and even trying out meetings with multiple different people.
7.4 Countermeasures
Schedule a series of portfolio backlog management meetings. Have at least two areas of focus: strategic and tactical. These can be two sections of one meeting, or two separate meetings. If there are multiple groups (containers) that work on related features, independently of other groups, yet they share a common interest in the strategic direction, then its best to have separate meetings. Have one overall strategic focus, to discuss strategy and direction, market changes, new ideas, customer updates, etc. One goal should be to identify new portfolio backlog items that might still be at the idea stage. Each group then has its own more tactical backlog meeting, where the primary focus is on moving features towards a ‘Ready’ state.
7.5 Actions taken
Set up a regular cadence of portfolio backlog meetings. Focus on moving portfolio items from “Idea” to “Ready”.
7.6 Intervention type
This is primarily a Container and Difference intervention. The backlog management meeting itself is a (often new) container. We need to ensure the right mix of significant differences – skills and perspectives – in the meeting.
7.7 Resulting benefits
The organizations that successfully adopt this approach find they have fewer delays, and increased overall coherence across the groups and teams working in the portfolio.
7.8 Signals of success or failure
Possible failure signs include meetings that grow too large, have non-contributing participants, or that never involve development team members. Other warning signs include getting to release planning meetings, and not having a good backlog of portfolio features that the team is already familiar with. Having the meeting dominated by status reporting rather than moving features towards ready is another warning sign.
8. The Relationship between Program Managers, Architects and Product Owners
8.1 Context
We use programs as a vehicle to deliver work. That work typically has defined targets for content and capabilities, and some expectations around dates. Our programs typically use three-month increments. Programs are expected to demonstrate tangible progress at the end of each three-month increment. Product managers are responsible for feature descriptions. Product Owners are responsible for user stories derived from those features. Architects define the technical aspects of the system, and define architecture work that needs to be done. Product mangers tend to be part of the portfolio team, working with defining features. Product owners tend to work with development teams, defining user stories. There are usually more product owners than product managers. The management of the portfolio backlog is the domain of the product manager; the management of the team backlog is the domain of the product owner. The two roles need to collaborate to avoid the emergence of handovers and delays.
8.2 Problem and Impact
It happens with many programs that user stories are not good quality. This is an aggregate summary of many of the problems we see in large portfolios with many programs and teams:
- Poorly written, vague, ambiguous, maybe even just a one-line phrase or sentence.
- No acceptance criteria defined.
- Not describing real customer value.
- No follow-up conversation happening between team members and product owners.
- No traceability between features defined by product managers, and the user stories defined by product owners.
- People writing a large amount of “user stories” that were neither stories, nor described any value for users.
- User stories that start with “As a developer …”. This is inappropriate, unless, for example, the team is developing tools or APIs for developers.
- Individual developers writing their own “user stories” that they work on alone. This is a symptom of people working individually, rather than as a team. There was no storming on user stories.
- User stories are really just task descriptions. While the tasks are important, they are not what a customer perceives as adding value. Too granular a set of “task descriptions as user stories” creates a lot of noise in a large program backlog, and encourages individualistic thinking and ownership rather than shared ownership.
- In some cases, individuals were writing user stories to capture activity such as holidays, meetings, and other things that add no customer value. This adds noise to the backlog, and contributes to a false impression of progress and productivity when looking at charts like Release Burn Ups or Cumulative Flow [9].
One of the root cause for many of these turns out to be product owners, product managers and architects not actively engaged in managing the backlogs together in a collaborative effort, and, following on from that, not facilitating effective conversations with the development teams.
8.3 Cynefin domain
Whenever there is potentially a cultural shift required I typically treat this type of problem as complex. From there, we were able to run some experiments. Eventually, we were able to create some recommended good practices, thereby moving the problem to the complicated domain.
8.4 Countermeasures
These are some countermeasure considered:
- Write personas, so that instead of everything coming from the perspective of a generic “user”, it becomes clear who is the recipient of the value.
- Have product managers and architects collaborate on portfolio-level feature descriptions.
- Provide training and workshops on defining customer value, writing features, deriving user stories, and taking those user stories through to demonstrable software.
8.5 Actions Taken
Have product managers and architects work together to define feature descriptions, and prioritize the portfolio backlog. Formalize the relationship between product managers, architects and product owners, creating subgroups where appropriate. These subgroups work together to provide a steady flow of good quality user stories from the portfolio backlog to teams.
8.6 Intervention type
This is first a Container intervention, and also a Difference intervention. We changed from having multiple containers with very little difference within them, to having different containers with a high degree of difference. The diagram in Figure 1 shows the before and after view, highlighting the containers, differences and exchanges.
8.7 Resulting Benefit
The feature descriptions improve, which also results in improved visibility into actual value provided. This in turn helps in conversations with customers and other stakeholders. Teams have a steady backlog of good quality user stories to draw from.
8.8 Signals of success or failure
Possible failure signs include deferring collaborative discussions between containers until release planning time. If teams are not continually coordinating and planning together there will usually be lots of avoidable churn. Another potential sign of failure is handoffs from product mangers to product owners to teams, rather than having collaborative conversations take place. Positive indicators of success include evidence for those collaborative conversations taking place, and a narrowing gap between customers and developers.
Figure 1 Before and After: From containers with the same roles to containers with a better balance of differences
9. Summary of Lessons Learned
This section summarizes some of the key lessons learned in managing portfolio backlogs. Some general lessons related to solving problems in organizations include:
- Understanding the nature of the problem helps us to take appropriate action to solve the problem. The Cynefin framework helps with this.
- Make sure you are solving actual problems and causes, not just symptoms. A3 problem solving helps with this.
- Understand how to create a balance between agility, self-organization and coherence. HSD and the CDE model helps with this.
- Focus on the end-to-end flow of value through your organization, and on actively removing anything that impedes the flow of work. Lean thinking helps with this.
- Understand what success and failure could look like before running your experiments. This will help you pay be selective about the patterns you pay attention to.
Some specific lessons related to managing portfolio backlogs in large organizations include:
- Define the focus of your portfolio. In general, it is good practice to base the portfolio structure on your product line rather than organization structure. The former is what your customers care about; the latter more temporal.
- Understand what content goes on the portfolio backlog. Define different types of items, e.g., features, initiatives, architecture items, etc.
- Focus on the flow of work from portfolio to teams. The portfolio backlog management approach is an enabler of flow. Define policies for centralized portfolio-level decisions and localized program- and team-level decisions.
- Set up a portfolio backlog management meeting at a regular cadence with the right participants. Create a Definition of Ready for portfolio items. Focus the meeting on feedback from the development teams, and on moving portfolio backlog items to a ‘ready’ state. Do not let it become a status or strategy planning meeting.
- Create conditions that encourage a strong relationship between product managers, engineering leaders and architects. Together they bring multiple important perspectives to creating the portfolio backlog items. Consider also adding user experience design leaders to this mix, depending on the nature of your products.
Finally, this is a process of continuous experimentation and improvement. While some things can ultimately be moved to the obvious domain of best practices, or the complicated domain of good practices, we still operate within an ever-changing and complex environment that requires continuous awareness, experimentation, learning and adaptation. We continue to experiment and make improvements.
10. Acknowledgements
Thanks to my manager at Cisco and our team for their support; and to the great teams and organizations that I have the privilege to work with at Cisco. Thank you Rod Nord for being my shepherd for this paper, and to Rebecca Wirfs-Brock for your help through the process – this paper would not have been completed without the support of you both.
REFERENCES
[1] D. K. Sobek and A. Smalley, Understanding A3 thinking : a critical component of Toyota’s PDCA management system. Boca Raton: CRC Press, 2008.
[2] D. J. Snowden and M. E. Boone, “A Leader’s Framework for Decision Making,” Harvard Business Review, November 2007.
[3] G. H. Eoyang, “Conditions for Self-Organizing in Human Systems,” Doctor of Philosophy PhD Thesis for Doctor of Philosophy in Human Systems Dynamics, The Union Institute and University, 2001.
[4] K. Power, “Social Contract Theory, Simple Rules and Self-Organization: A Perspective on Working Agreements in Agile Teams,” presented at the Agile Processes in Software Engineering and Extreme Programming, 15th International Conference (XP2014) May 26-30, MMXIV, Rome, Italy, 2014.
[5] E. E. Olson and G. H. Eoyang, Facilitating Organization Change: Lessons from Complexity Science. San Francisco: Jossey-Bass/Pfeiffer, A Wiley Company, 2001.
[6] Cognitive Edge Web Site. Available: http://cognitive-edge.com/
[7] D. Snowden and T. Quinlan, Cynefin and SenseMaker Course Material: Cognitive Edge, 2014.
[8] K. Power, “Definition of Ready: Experience Report from Teams at Cisco,” presented at the Agile Processes in Software Engineering and Extreme Programming, 15th International Conference (XP2014) May 26-30, MMXIV, Rome, Italy, 2014.
[9] K. Power, “Metrics for Understanding Flow,” presented at the Agile Software Development Conference (Agile 2014), Orlando, FL, USA, 2014.
