18F's projects result in tangible outcomes, like redesigned websites and more efficient government services, as well as outcomes that are less easy to measure — like training or coaching agency staff in running essential technology product work.

To measure the output of these less tangible outcomes, we use tried and true techniques — drawing from tech, management, and other industries — to make our work visible and demonstrate how we're serving the American people.

Training product owner skills

During one recent 18F project, our goal was to enable agency partner staff members in being first-time product owners. None of the agency staff had ever done product work, but they were tasked with managing a complex digital services project. It was important to the agency that we teach them well.

How would we know we were successful in turning agency staff into successful product owners?

Adding a measurable framework

There's an adage that "You can't change what you can't measure." We needed a way to measure the impact we were having, so we could know how well our coaching was working.

We decided to measure all the product owner trainees' progression in a set of important product owner skills. We brainstormed all of the product owner skills we could think of, then de-duplicated the list. Next, we voted on the skills using n/3 voting — that is, each person got one-third as many votes as there were stickies. For example, if there were 30 stickies, each person would get 10 votes.

In the end, we decided to measure product owners' progression in a set of 10 key skills, including Product discovery, Setting goals and a roadmap, and Managing vendors.

The next problem to solve was, how would we measure the development of these skills?

We decided to use a skill development approach called EDGE, which stands for Explain, Demonstrate, Guide, Enable. This is a decades-old, tried-and-true methodology how to teach a skill.

First the teacher Explains what they're going to teach, then they Demonstrate it by showing the skill. Next, the learner starts trying it on their own, but the teacher is there to Guide the learner, correcting or refining the skill as they practice it. Then, the teacher Enables the learner to take ownership of the skill — the learner starts doing it on their own, intervening only when the learner requests help.

We decided to assess product owner skills by asking each apprentice product owner to do a self-assessment of their skills every few months, using a survey based on EDGE.

We created a table with the EDGE stages across the columns and the product skills down the rows. At regular intervals throughout the project, we asked the apprentice product owners to assess themselves on each of the 10 product ownership skills. Were they unfamiliar with the skill and needed explanations or demonstrations? Were they doing the skill, but still in need of guidance? Or did they think they were enabled to fully own the skill?

Need explanation Need demonstration Need guidance Fully enabled
Product discovery X
Set goals and roadmap X
Manage vendors X
Early skill development self-assessment. Shows the self-assessment of an apprentice product owner who needs a demonstration of product discovery, an explanation of how to set goals and roadmap, and guidance in managing vendors.

Our goal was to get all the product owners to "Enabled" in all 10 skills. When we ran self-evaluations, we would look at what skills were more to the left of where we expected, and focus more on supporting the product owners in learning those skills.

We produced a new chart for each self-evaluation, so we could look back and see changes over time. As the product owners developed in skills, we could see movement to the right, toward the "Enabled" column.

Need explanation Need demonstration Need guidance Fully enabled
Product discovery X
Set goals and roadmap X
Manage vendors X
Later skill development self-assessment. Shows the self-assessment of an apprentice product owner who is further along in skill development than the last table, needing guidance in goal-setting but enabled in all other skills.

By using this method, we were able to see that we effectively coached these product owner skills, and left the agency in good shape to manage their technology modernization project.

Using multiple types of assessments

In this project, we only used the trainees' self-assessments. Next time, in addition to self-assessments, we'll ask the 18F leads to also assess the apprentices' skills, using the same EDGE chart. Then, we'll compare the self-assessment to the coach assessment and talk about the similarities we see, and the differences we see.

With both assessments, we can have two data points instead of one, which we expect will give a clearer picture of the skill development.

In conclusion

You can't improve what you can't measure. When a digital services project includes less tangible outcomes such as skill development and coaching, make them visible by finding metrics to measure, and measuring them over time.

By measuring product owner skills over time, we were able to see that we were effectively training our agency partner's staff, and that we were delivering value by supporting our partner agency in sustainably managing their technology modernization effort.