Math by Tom Brown Full

At Substantial, we work to continuously improve the process by which we collaborate with our clients to build awesome products. One of our latest concerns is how we can integrate validation into our product development life cycle in a way that works well with our clients. As part of this effort, we held a brown bag panel discussion on the topic, with a focus on the challenges of utilizing validation in a client services business. The multidisciplinary panel consisted of leads from several projects at different points in their life cycle, each currently utilizing validation in some form or another. Below are some of the major highlights from the discussion.

What is validation?

First, a definition – “Validation: the assurance that a product, service, or system meets the needs of the customer and other identified stakeholders.”

Projects that have clear up-front goals and validation are BFFs

One of the big takeaways from the discussion was that the existence of upfront goals enables a project to utilize validation much more effectively. When we start a project with a client that has clear and measurable goals, we are best set up for success. We can quickly align on these goals, helping us focus on what we must build to achieve them, and what we must measure to validate that we are building the right thing. Throughout the project, we can return to these goals to guide our efforts.

On the flip side, all sorts of trouble can ensue when we start a project with less defined goals. It is more difficult to establish which features are critical for success during prioritization, leading to a longer development phase. Additionally, because the goals are not well understood, it is challenging to come up with measurable success criteria to use to validate the product and its features.

We are actively working to improve our ability to help our clients define goals for a product. Everyone’s natural tendency when they have a great idea is to run out and immediately build it (that’s the fun part), but taking a bit of time up-front to ensure the goals are understood and measurable leads to a better chance of success in the long run.

Validation beyond metrics gathering

All too frequently, we view validation solely through the lens of metrics gathering. Although gathering metrics is an important tool to use in product validation, when we only concern ourselves with metrics, we will frequently represent the validation work we need to do as a single “metrics” task or story in our feature backlog. When we do this we miss out on opportunities to learn and we enable decision makers to keep deferring this task. If instead we view validation more broadly, including more techniques such as customer interviews and user testing, we see that validation is a tool that we can include at all phases of product development.

A holistic view of validation requires you to build validation into your process. Instead of viewing it as a task that needs to be checked-off before launch, think about how different validation techniques can be applied with everything you deliver: Starting an epic? Do some user interviews. Getting ready to move a design into build? Do some user testing with lightweight prototypes. Starting to develop a feature? Identify which metrics need to be present to allow for measuring the success of the feature.

In addition to adding validation as part of the different hand-off points in your process, you can also bake in validation exercises as part of your overall weekly process. For example, you can block out 2 hours on Wednesdays to do user testing as a team. By setting aside that time, you start making these activities habitual, and stop thinking about them as chores.

Releasing enables validation

Another interesting topic discussed was how the act of releasing the product to a broader audience is a critical enabler of validation. There is a natural resistance to show off your work until everything is smooth and polished. However, having a product in the wild makes all sorts of validation techniques so much easier (or possible) to perform.

You can spend a lot of time theorizing about the ideal, or you can get an initial release out there and start validating with real data. For example, if your product’s conversion flow is critical, you can invest months of up-front time trying to get it just right, or you can release it and start measuring and optimizing. Before your product has been released, determining whether a specific feature will enhance engagement can be a bit of a guessing game. After release, you will have a list of customers you can interview, or you can expose marketing copy for the feature and see if anyone actually interacts with it.

Just like you have to build validation into your process to utilize it to full effect, you must also tune your process to allow for rapid release if you want to maximize your opportunities for validation.

The topic of validation is vast, but the return on investment is high. As we validate our experiments in validation in our client services business we will continue to share our experiences with you.

Main image from Flickr user Tom Brown.