Making early and meaningful contact with customer feedback requests dramatically improved our product velocity, employee engagement, and revenue retention. By performing Product Discovery tasks in the Customer Success / Solutions Engineer role, I helped: buy information in the most economical way, accelerate learning to reduce product feasibility and usability risks, resolve the cognitive dissonance demotivating the team, reduce cycle time to less than 2x weeks, and hit our revenue retention goal.
1. INTRODUCTION
As a Solutions Engineer, I’m thrust into situations to handle customer relationships that have rapidly deteriorated. In my experience, there is a high probability that customer relationships have soured because there has been inactivity on critical feedback. Perhaps a useful idiom to describe this scenario would be dancing with our hands tied. In this context, it describes a situation where one is accountable for the outcome yet they are not given any authority to make decisions and take action.
For this specific experience report, I was fulfilling this role in a tech startup. I had the responsibility to ensure customers are successful with using the SaaS solution and renewing their licenses. In any conversation or engagement with these customers, I would be constantly receiving their feedback, mostly in an unsolicited manner. If you listen, they will tell you where there is friction, where there are problems, where there are opportunities. That maps directly onto the Continuous Discovery or Product Discovery models.
Customer feedback is a primary feedback loop in product development. So, what generally happens with that feedback loop? In my humble and limited experience, the feedback specifically collected by customer-facing teams is at best added to a spreadsheet, and then handed off to another team. It’s fair to say that the customer would expect that this feedback has a lifecycle, it goes through some formal process, and a decision will be communicated. The reality, again just in my experience, is the feedback is just left as an open item. It remains in an ambiguous state, and at the same time, the direct customer-facing team (i.e. myself as the Solutions Engineer) remains accountable for managing customer expectations. There is nothing fun about that part of the job. Furthermore, this situation causes a cascade of other issues in the company: what if the customer’s feedback (needs, pain points, desires) is a source of innovation; what is the cost of all the chaos that ensues across sales, marketing product, finance, and other teams about revenue retention and growth related to the other work that got delivered that did not resolve these customer requests; even worse yet, what happens to employee engagement when people’s rewards and incentives are measured on customer satisfaction and renew rates? All of this is outside of your control if the friction is related to a lack of product capabilities.
Without going off on a tangent about these 2nd, 3rd, or 4th-order consequences, this experience report just focuses on how to effectively manage direct customer feedback. Specifically, direct customer feedback that is received by customer-facing teams. How to manage customer feedback is a problem, and to support that statement I would just look to the market to see all the new products emerging on the market today (Jira Product Discovery, ProdPad, Productboard, Cardinal). All are targeted directly at solving the intersection of customer feedback and product management.
Product Management seems to be a battle of the frameworks too, and customer feedback gets overshadowed by Product Requirements Documents (PRDs) and product roadmaps full of internal stakeholder features. I have drawn some insights from this experience based on that Product Management pattern, and I share those in a special section of the report. Is it any wonder a common expression from customers is that ‘all feedback vanishes into a black hole’.
In this experience report, I’m going to cover the benefits of performing Product Discovery activities on customer feedback, and treating customer feedback as the primary control loop in Product Development when the company has passed the Early stage (Product-Market Fit) and in the Growth stage. I will use the Principles of making early [and meaningful] contact with the problem space, the importance of keeping feedback local, buying information at the optimum time, and the principle of inactivity. I will also showcase how successful companies utilize prototyping to improve their product velocity.
2. Background
This specific experience is working within a startup in a role that encompasses responsibilities across the domains of Solutions Engineering or Customer Success. The startup was not in its early stages, rather, it was in its Go-To-Market stage, needing to continue to grow in order to reach critical mass in the market. Naturally in this stage retaining key large customers that have invested and adopted the SaaS product at scale is supercritical. They are after all funding the business through annual recurring revenue (ARR), and product usage.
So when I arrived at the company, in the Solutions Engineer/Customer Success role, customers immediately flooded me with feedback. In the Continuous Discovery model[1] terminology, this feedback may be an opportunity that translates to an unmet customer need, pain point, or desire. At this point in time, the level of engagement from the customer was very high, as they would openly share what they needed to fully adopt the solution across the enterprise. That is a gift of their time and creative thinking. Yet, I observed their level of engagement dwindle considerably, as their level of interaction was met with inaction on our part.
The product we deliver to market is a Collaboration tool, designed to enable the right human behaviors and practices for teams as they plan and deliver good products. This is what attracts me to such a solution; it is a tool that can unleash some of the untapped human potential of individuals that work in a team or large programs. The product, however, is highly dependent on integrations to 3rd party ALM solutions. This presents a double-edged sword in the sense that solving for the 3rd party integrations, while it is a unique differentiator and helps with the barrier to entry, it also presents a whole slew of product and technical challenges and risks. Integrations can become a slippery slope; scaling and sustaining multiple integrations over time requires active product leadership to display prudence and discipline. The team did this well from the start but if we chart this on a spectrum maybe now either too much or too little of both of these leadership traits were stymying our efforts to expand our customer base.
Therefore, the key to this experience report is a specific customer [feedback] request related to a 3rd party integration. The 3rd party vendor had recently improved their market offering, and the customer was requesting that our [integrated] solution fully support the new business processes ushered in by the new vendor offering. By the time I am involved in this scenario, this specific customer request is outstanding for over 18 months. Keep that in mind, they had discussed this request ad nauseam by this stage. To get an idea of the impact or scope from the customer’s perspective; at scale this was affecting approximately 290 teams that regularly plan and work together at scale. As you read on, the technical risk that I discovered was the 3rd party vendor was only providing an unsupported API. We will make sense of the complexity of all these elements using the Cynefin framework[2] in the story below.
3. The Story: Treating Customer Feedback like a Product Backlog
3.1 The Problem: Handoffs, Delays, And Disempowered Stakeholders
In my first meeting with one of our most high-profile customers I learned that their highest priority request had been aging for more than 540 days. The request had been communicated in meetings, emails, spreadsheets, etc., yet that was the extent of it. Simply put, the request had been discussed ad nauseam. There was palpable tension during the discussion, but also a glimmer of hope perhaps because I was the new face to talk to.
In the beginning, the team had done some very preliminary analysis by browsing the vendor’s public-facing documentation and concluded that there was no interface or mechanism to build an integration. That’s where the discussion stopped, and it was implicit. There was no communication with the customer about any definitive decision. There was no communication or response that showed we had analyzed the request based on product viability or feasibility and had reached an explicit conclusion. The request just remained open on a spreadsheet, frequently rehashed in meetings. We knew the psychology of long queues on teams/people, it is demotivating; the inactivity on this request was having the same effect.
At this point, there was a lack of clarity, just ambiguity, between all stakeholders about what happens next. To recap, we had only done inconclusive preliminary research, which led us to make incorrect assumptions, and no formal decision had been made. I even try to encourage the Product team to formalize strategic product artifacts like a product mission/vision, goal, and definition. We could then evaluate the request/s in terms of product viability and feasibility, and use that to facilitate the conversation with the customer and reach a conclusion. That activity did not transpire either.
A few weeks passed, and I observed the conversations with the customer becoming very terse because all that transpired in every meeting was a rehash of that list of requests in the spreadsheet. Yet again, my job was being reduced to reviewing rows in a spreadsheet. I focused on gaining trust, and trust is gained by doing what you promise. I eventually gained enough trust based on my delivery against a myriad of enablement activities, and they started to share information about their internal state of affairs. Internally they had some business units making threats to use other competitor solutions because of the utility of purpose. Note: the competition solution did not have a market offering for this integration either. We quickly engaged in an open conversation about them not renewing their contract, which would result in a loss of critical ARR. These are tough conversations for a newly hired Solutions Engineer/Customer Success representative to have to manage.
How did we get to this place? Why did we get here? The real problem, as I saw it, was that we failed to manage the customer feedback in a meaningful way, and in a timely manner. The distinction between the two different dimensions is important. There is an optimum timing for handling feedback, and it’s about treating the feedback in a meaningful way, not to dismiss it.
The request had been handed off between different teams/roles, but there had been no real meaningful discovery or experimentation to analyze the options related to this request. Worse, the continuing conversations with the customer lacked detail, and no decision was being made. Now as the Solution Engineer meeting with the customer regularly I have to manage this situation but I have to do this figuratively with one hand tied behind my back because I’m not in a position of authority to resolve the situation. Yet if you’re a Theory Y[3] type of knowledge worker, then you want to take the initiative to manage this in a meaningful way, and get to a viable outcome for everyone.
In any respect, no one was making a decision about this request, it was a stalemate situation. I realized quickly that it was the confluence of a few factors at play, such as a lack of discipline to analyze feedback in a methodical way; the perceived large batch size of the request becoming a demotivating factor too. This is common as large batches do demotivate people.
One departing thought, before we dive into how I was able to deliver a successful outcome, is on the notion of collecting customer feedback yet not treating it as a closed loop. As a team, we were doubling down on gathering customer feedback via surveys, interviews, and by increasing the number of touch points with customers. All these activities need to be encouraged as they are ways to communicate and collaborate with your Users and collect feedback. Yet, spending so much effort to collect this feedback, and failing to action it correctly or close the feedback loop, is akin to the fallacy of measuring activity vs. outcomes and declaring success.
3.2 Optimizing Customer Feedback using Product Discovery Techniques:
Being in this customer-facing role, my value proposition was to ensure customer feedback is a closed feedback loop. That means a formal decision or response is communicated to close the loop. I desired an outcome where we would either formally communicate that we would not be acting on the request or propose that we would build and deliver a solution to the market. d Guided by that desired outcome I now had some scope for my activities, and I set out to learn if a solution was viable, and if so, prototype something to get buy-in from everyone on a way forward.
To clarify, the outcome I was seeking was not rocket science. I think it is just practical and polite when a customer gives you detailed feedback to reciprocate in some manner. It is generally the case that you both have the same objective in mind: to expand the adoption and usage of your product in earnest.
I conducted a collaborative whiteboard session to discover the Use Cases that this integration would support. What were the manual or tedious business processes that were creating friction in their organization, and what the new integration would help solve? That whiteboard session was immediately beneficial. It took about an hour and at the end, I had now three definitive Use Cases, and a business priority for each Use Case. This request now had some shape, and very clear [business and technical] logic about to break down the request into smaller batches. I was exercising the principle of making early contact with the problem. Yet, it is not sufficient to only make early contact, you must make early and meaningful contact with the problem/opportunity.
Now that I had some focus guided by the three Use Cases, the immediate next activity I knew I could do was to research, test, and experiment with any API interface the 3rd party product provided. I set out doing just online research and finding any public-facing documentation online. I also tried to request a trial environment of the 3rd party solution, as performing actual tests in a trial environment would further validate any assumptions and provide higher fidelity than just reading documentation. If we look at this through the lens of Information Theory; I would reduce risk by reducing the uncertainty about the original assumptions the team had about this request. The team had originally undertaken some very preliminary research online and concluded there was no public API available, hence the request was not feasible.
Quick recap, at this point I had just performed what most would consider to be design or user research interviews (the Why and What) to map out the opportunity space in detail. The initial discovery work is just the online research. Since our existing integrations leverage a REST API architecture, my online research focused directly on the 3rd parties REST API. I could immediately confirm what the team had previously already concluded: while there was a published REST API, and it was extensive, there was no documentation for the specific endpoint that would be required to fulfill the 1st their highest priority Use Case.
This problem was that this online research was insufficient because the information online suggested that the API provided by the vendor solution was insufficient to meet the customer’s highest-priority Use Case. However, what I perceived was the idea that if there was a REST API available but a specific endpoint was not available, then it stood to reason that it was worthwhile validating the missing API endpoint directly with the vendor. After all, the information online might give us some direction but it’s not a substitute for learning via a conversation. Remember the agile manifesto value of Individuals and interactions over processes and tools, in this case, online search.
Getting a one-on-one conversation with people at large top Fortune 500 companies requires some creative thinking. The path of least resistance to me stuck out to be networking with the 3rd party vendor account team that was engaged with our shared customer. Unfortunately, they just used the standard operating procedure and directed me to use the formal support channels to ask my question about the missing API endpoint. So, I submitted a question through their formal support site. The response that I received was the standard formal response, it just reiterated what was available online in the published API documentation. I was unsatisfied, though not at all surprised at the standard response to the question I had submitted to their Support. At this stage, my discovery work has not uncovered any new information to validate or invalidate our assumptions. The next steps I undertook to explore options or gather new information were as follows:
- First, I wanted to get access to a trial environment and explore the API first-hand. The ability to get a trial environment provisioned needed to be requested and facilitated directly with the vendor account team as there was nothing publicly available online.
- Second, I engaged their User Community and posed the question about the REST API and absent endpoint in their Community forums.
In a matter of hours after I posted the question in the vendor’s User Community, I received a response from a Community member (an SME at the vendor company) and what I learned was that there is an undocumented and unsupported API endpoint for the resource we needed to integrate with. This was new information that now changed any decision about the complexity or effort required to deliver a solution.
After I received access to a trial environment, I used Postman to access the REST API and explore their support and unsupported API endpoints. The unsupported endpoint was without documentation, but using the existing documentation and Postman I could explore the unsupported endpoint and document all the Request Methods and Response Formats available.
At this point, I was ready to build a prototype and run experiments. The prototype focused on the data layer, data mapping, and data flow between the two systems. A Postman Flow was used to build a high-fidelity prototype that simulated the synchronization of data between the two systems for the given Use Cases. I documented all the data inputs/outputs and transformations that would be needed in the data flow so engineers would be armed with the API endpoint, request methods, and response formats to code for when it came to building something.
What should be glaringly obvious is that customer feedback and product backlogs are linked queues. So, operating from a First Principles mindset, I took the initiative to manage the customer feedback as a backlog. I started to undertake these informal Product Discovery activities on the customer feedback items based on the principle these are linked queues and long queues are bad.
My objective was to gather first-hand data to help Product Management with decision-making when we use tools like the Impact-Effort matrix for potential new Backlog items. As it relates to Customer Feedback specifically, I believe decisions are made based on incorrect or implicit assumptions. So, gathering real data (buying information so to speak because you spend some effort to do this), and making better decisions about the customer feedback is something that can have compounding benefits in Product Development. Another goal is to eliminate all the risks associated with long queues, and since this customer feedback queue sits upstream of the product backlog [queue], then the benefits that flow will be compounding or exponential for the whole system (all teams).
3.3 The Result: Improved UX, Improved Flow, And Revenue Retention
After I demonstrated the high-fidelity prototype to both the customer and engineering we then moved to building an MVP. Since we were relying on an unsupported API endpoint, we wanted to be prudent about adding new code to our code base. We ended up building and releasing scripts that would fulfill all the requirements of the first critical Use Case. The cycle time to complete that work after it was pulled from the Product Backlog and delivered was less than a single 2x week Iteration. The high-fidelity prototype helped speed up the design, build, and test cycle because the information about how the API endpoint behaved had been documented. We went from >18 months of inaction to just a single Iteration to deliver working software, the litmus test of value.
At the moment we have kept the shipped product as a script because it is built on an unsupported API endpoint. The desire from the customer is that we add this new functionality to our product code base. We know the risk that this presents because it is officially an unsupported API from the 3rd party vendor. Yet we can at least now have a bonafide conversation about the risks, and costs associated with the request because we have gained trust by taking some action on the feedback. The conversations now are at a level of detail that is meaningful and that is creating generative conversations.
In the process of acting on this request in a meaningful way we have made the product more viable and feasible for another customer segment. More importantly, we are slowly regaining trust as the preferred solution across the whole enterprise for this key customer. When there was no action on that feedback some divisions had started deviating from their enterprise standard and using other software tools.
Another result from this experience is that I’ve gained new skills, deeper working and collaborative relationships with the engineering team, and an immense sense of job satisfaction too. Furthermore, it’s evidence that you improve product velocity and cycle time by creating high-fidelity prototypes during the discovery phase[4]. Alas, it is good to mention the customer renewed their licenses, and the company’s cash flow and revenue were not impacted.
I had essentially created a very regenerative fast feedback loop at multiple levels that allowed for effective decision-making at the product, customer, and business levels all by making meaningful contact with the problem.
4. The Learnings: What to Keep Doing More Of
In regards to managing customer feedback, there is both optimal timing and optimal action; get these right and you optimize the economic payoff. The most important learning though comes from the need to take meaningful action with any problem or request [that constitutes the feedback]. In this situation, the assumption was the request was too much effort and too much complexity versus the benefit. When feedback loops—in this case, the primary customer feedback loop—have long closure times various other factors come into play that can have severe economic consequences.
Customer feedback needs to be treated as a pseudo backlog, and thus subject to some relative level of continuous discovery activity to be managed properly. Remain guided by queueing theory as customer feedback is a linked queue in product development. Customer feedback is an input [linked] into the Product Backlog, yet in many scenarios, it seems there is no real Owner of the Customer Feedback queue. Perhaps that lack of true ownership leads to neglect, oversight, and mismanagement. If you are interested in flow metrics like lead time or cycle time, exploiting opportunities when they present themselves, and leaning more towards the agile value of Customer Collaboration vs. Contract Negotiation, then manage customer feedback in a robust way. In no way shape or form does this mean you need to deliver new product functionality based on every feedback item? On the contrary, the requests must still be evaluated to be viable and feasible. Yet, in that process, you will have engaged with the customer in a meaningful way and built a more intimate relationship just because you listened and closed the feedback loop regardless of the decision.
We learned that there are benefits to extending Product Discovery or continuous discovery habits and activities to customer-facing teams. We can buy information in a very economical way, it is also an opportunity to allow people to take initiative and perform work that they are interested in that benefits the whole organization, and it’s a chance to allow people to upskill. The benefit of that is increased employee engagement.
Another learning is that keeping the feedback local ensures the customer-facing team can have more control over the situation. It seems logical that since the Customer Success or Solutions Engineering teams are dealing directly with the feedback it makes sense to empower them to manage the feedback. As it is typically done today, the feedback is handed off to Product teams, leaving the customer-facing teams to just act as a proxy. Important details or context gets lost in this process if there is not a formal hand-off and Product teams focus more on their product strategy process which oftentimes is separate.
It seems very effective to use my capacity in the Solutions Engineering/Customer Success role to essentially do the discovery work without detracting from the product team’s capacity, and validate the feedback earlier so we could either pursue or truncate a bad path quickly. We constantly need to buy information in product development through experiments, spike work, etc. In this scenario, we utilize the capacity of other people/teams to speed up learning through prototypes and that has tremendous benefits of speeding up the product development lifecycle overall.
Below is a summary of what I have learned during this whole experience by practicing some of Donald Reinersten’s Principles of Product Development Flow:
- There is a cost of delay with customer feedback. It is essential to reduce the amount of feedback in progress.
- Make early and meaningful contact with critical feedback. This requires a commitment of time and effort to learn using a hands-on approach. Allow teams local to the feedback to use their capacity to do the discovery work..
- Feedback is highly variable and random, thus managing it locally with the Customer Success or Solution Engineers is the most economical way to gain the right information to validate feedback.
- Encourage Customer Success or Solution Engineering teams to take initiative in handling the feedback they receive, this is a great way to upskill people and tap into their intrinsic motivation.
- Expand these roles [Solutions Engineering and Customer Success] to be Prototypers. In Marty Cagain’s words using the Product Manager, Designer, and Prototype triad, ‘provides a different approach to new product creation, one that costs dramatically less and is much more likely to yield the results you want.’
- High-fidelity prototypes can just be data flows, showing the inputs and outputs between systems. Modeling the data flow is most effective vs. UX wireframes etc. The information contained in them will feed forward into the product development process.
- Making Product the gatekeeper of customer feedback might be the most expensive way to handle feedback.
- A closed customer feedback loop will build trust and a resilient customer base.
- Once product market fit is achieved, and you are in the product adoption phase, customer feedback should be weighted just as high as other key product strategies.
5. Industry Insights
When I reflect on this whole experience the concept of Burn the Backlog comes to mind. Customer feedback is simultaneously timely and the ideas originate based on the gap between their current experience and their needs. One could surmise that it stands to reason that prioritizing your Product Development on the customer feedback requests will result in better outcomes. The feedback will give you insights into what the customer regards as adding value—something you will pay for—or something that will increase usage.
It is not the concept of a Backlog that is the issue. It is how we use Backlogs that causes the anti-patterns. Sometimes there is a lot of time and effort spent creating a Product Strategy that has a 1-3 time horizon, has a hierarchy of backlogs, and at the lowest level is an internally focused list of Features (i.e. akin to the Feature-Model, Project-Model, or IT-Model). Backlogs have turned into long queues of large batches of work that are internally focused, reminiscent of PRDs, defined upfront and too far in advance, and focused on output, not outcomes. The outcomes we need are the delivery of new functionality that solves a current problem, not just delivering something that’s cool.
In this particular circumstance, performing some formal Product Discovery activity on the customer request was pertinent to retaining revenue and improving the customer experience which impacted hundreds of teams. Also, it has the potential to open a new market segment for the product while it is in its growth phase, all with little to no downside or risk. After prototyping that request still ended up in the backlog (little ‘b’) but it was timely, desirable, viable, and feasible—validated with a true lean business case.
Another key insight that I have learned since documenting this experience, is that companies like Ramp (fastest growing SaaS company to 100 million ARR), Tesla, and SpaceX, all work in this way: make early and meaningful contact with the problem and use prototypes. They place just as much emphasis on Product Discovery as they do Product Delivery because they are trying to deliver better products to market by solving problems, not just delivering a bunch of stuff [Features]. They empower the teams to solve problems or innovate to exploit emerging opportunities. The concept of empowered teams is everywhere. They do all this while being directionally aligned with the product vision, strategy, and immediate objectives.
6. Acknowledgements
Thank you to the Lean-Agile practitioners everywhere, especially Don Reinersten, for imparting the principles of Lean Product Development to me through your content. It has inspired me in the work I do and given me purpose. I specifically want to acknowledge my Experience Report shepherd Joe Yoder. His ability to quickly glean the details of my experience give me guidance on how to structure the report, and learn together through the dialectic process is invaluable and a big confidence boost.
REFERENCES
Reinersten, Donald G. “The Principles of Product Development Flow”, Celeritas Publishing, 2009
Bogsnes Bjarte, “This is Beyond Budgeting – A guide to more human and adaptive organizations”, Wiley, 2023
Benefield, Gabrielle. Mobius a Radically Simple Idea, 2011 Blog Post
Cagan, Marty. “Startup Product Management”, SVPG, June 2005
Cagan, Marty. “Rapid Response”, SVPG, February 2006
Torres, Teresa, “How Continuous Discovery Works (and Doesn’t) in Early-Stage Startups”, April 2022
Lenny’s Podcast. “Velocity over everything: How Ramp became the fastest-growing Startup ever | Geoff Charles VP Ramp”
[1] Teresa Torres writes about the Continuous Discovery language and habits on her website Product Talk. Torres, Teresa, “Everyone can do Continuous Discovery” August, 2nd, 2023. [ref].
[2] The Cynefin Framework is a conceptual framework used to aid decision-making in complex environments, developed by Dave Snowden in the late 1990s.
[3] Theory Y, was developed by organizational psychologist Douglas McGregor. Theory Y knowledge workers are seen as capable of taking responsibility, exercising self-control, and directing their efforts toward achieving organizational objectives. In contrast to Theory X, which assumes workers are inherently lazy and dislike work
[4] Ramp has mentioned that high-fidelity prototypes have helped it achieve greater [product] velocity. Ramp is considered to be the fastest-growing SaaS company based on reaching $100 million in annual revenue in a two year period.
