How to connect product OKRs to public value when ROI is not the point
In 2011, I wrote a short paper about e-commerce KPIs.
The route from product activity to business value was relatively visible. Attract suitable customers, help them find what they need, convert more visits into purchases and give customers a reason to return.
Revenue did not explain everything, but there was at least a recognisable transaction at the end of the journey.
Much of my more recent work has been with data products for research. Here, there is no sale at the end and often no sensible financial return on investment to calculate.
The longer-term hypothesis is that better access to high-quality health data can support better research, which may eventually contribute to improvements in healthcare. But a product team cannot claim that releasing a dataset directly improved a patient’s outcome.
So how do you set meaningful product OKRs when revenue is not the objective and the social value may take years to emerge?
Start by separating KPIs from key results
This sounds obvious, but the two are often treated as interchangeable.
A KPI tells us how part of a product, service or organisation is performing.
A key result describes a measurable change we intend to achieve within a defined period.
Data-release lead time is a KPI.
Reducing the median data-release lead time from 12 weeks to 4 weeks by the end of the year is a key result.
The distinction matters because a measure does not need a target to be useful. It may help us understand what changed or spot when progress in one area comes at a cost elsewhere.
If release speed is a key result, for example, data quality, privacy and traceability may need to act as guardrails. Otherwise, the team could technically meet its target while making the product worse.
Begin with the value, not the available measures.
In a commercial product, the objective might be profitable growth, retention or reduced cost to serve.
In a public-interest organisation, the objectives are different. They may include enabling valuable research, broadening access to data, improving the representation of different populations or protecting public trust.
Take a research data product. The organisation may want to enable more valuable research using health data, but that is too broad to guide a product team. We need to be clearer about what should change for researchers.
Can they find the relevant data? Can they judge whether it is suitable? Can they build the cohort they need without weeks of manual support?
Those questions lead to more useful key results, such as making priority data available sooner or reducing the effort needed to build a cohort. Quality, privacy and security remain constraints on how we achieve those results, not trade-offs we make to reach them.
This gives the product team something to act on without pretending to control the eventual social impact.
Follow the evidence beyond the release.
A common weakness in public-value measurement is that it stops at delivery.
The team reports how many datasets were released, how many fields were processed or how many users received access. These measures describe activity and scale. They do not show whether anyone was able to create value from what was delivered.
1. Did we deliver what we said we would?
This is where I usually start because it is what we can see most clearly. We know when the data arrived, when it was released, whether the pipeline ran cleanly and how much manual work was needed along the way.
But hitting the release date only tells me that we delivered. It does not tell me whether researchers could do anything useful with what we released.
2. Could researchers find and understand it?
The next stage is whether researchers can make sense of what has been released.
I would first look at whether the high-value fields are clearly defined and whether researchers can see where the data came from, how its quality has been assessed, and what its limitations are.
The better test, though, is what happens when researchers try to find something. How long does it take them to identify the right dataset? Can they complete common discovery tasks without help? The questions they still have to ask often tell us more than the documentation we have published.
The number of data dictionaries published would be an output. Whether researchers can use them to understand the data is the outcome.
3. Was the data suitable for the intended research?
Even when data is available and well documented, it may not be suitable for the research.
The real test comes when researchers try to use it. Can they construct the cohort they planned? Do projects have to change scope because important data is missing? Are critical gaps only discovered after access? We also need to know whether the data covers the populations, conditions and time periods the research requires.
This is harder to measure than the number of datasets published, but it gets much closer to whether the product is solving a real problem.
4. Was the data used successfully?
The fourth stage looks for evidence of actual use.
Possible indicators include:
- Number of active projects using the data
- Repeat use of the same data across different research questions.
- Time from access to the first successful cohort or analysis
- Number of projects progressing from feasibility to analysis
- Completed analyses, abstracts or preprints acknowledging the data resource
These are stronger signals than access counts or downloads. They show that researchers were able to do something with the product.
They still require interpretation. One high-value study may create more public benefit than dozens of low-impact uses.
5. Did that use contribute to wider value?
The final stage is farthest from the product and hardest to attribute.
At this point, the evidence becomes less tidy. We can follow the work into published papers, citations, further use of the research, new collaborations or funded studies. Occasionally, there is a clearer trail into clinical guidance, public policy or changes to services and treatments.
None of these outcomes belongs to the data product alone. Taken together, however, they show whether the data contributed to something beyond its use on the platform.
These are not product delivery metrics. Nor do they prove that the data product caused the eventual outcome.
They provide evidence that the product contributed to a chain of activity that created wider value.
A mixture of quantitative measures and a small number of well-evidenced case studies may be more honest here than trying to reduce every form of social value to a single score.
Be explicit about control and attribution.
The measures across the chain should not all be reported in the same way.
The product team can largely control release processes, documentation and product functionality.
It can influence whether researchers find the data suitable and can use it successfully.
It may contribute only to publications, policy changes, or improvements in healthcare.
That distinction prevents the team from either overclaiming distant outcomes or retreating into delivery metrics because they are easier to count.
A credible product story might therefore say:
We reduced the time required to release priority data, improved researchers’ ability to identify suitable cohorts and saw increased use of the data in completed analyses. Several resulting studies have since been published, furthering research.
It should not say:
Our data release improved patient outcomes.
Use a small number of connected OKRs
A product team does not need an exhaustive cascade of targets.
| Organisational objective | Product outcome | Example key results and guardrails |
| Accelerate valuable research | Researchers can reach usable data sooner | Reduce release lead time; reduce time to first successful cohort; maintain quality and access-control standards |
| Improve the usefulness of data | Researchers can determine whether the data meets their needs | Increase successful cohort construction; reduce late discovery of critical data gaps; maintain clear provenance |
| Protect public trust | Data is used safely, transparently and for agreed purposes | Apply access and consent decisions within agreed timescales; maintain complete audit information; monitor material incidents and researcher experience |
The remaining measures can sit underneath these as operational or diagnostic indicators. They do not all need to become key results.
Some measures require judgment.
Terms such as “appropriate reuse”, “representative data” and “known limitation” can look measurable while hiding important decisions.
Appropriate according to which policy? Representative of which population or research question? When is a limitation significant enough to record?
These are not questions that a dashboard can settle.
Each important measure needs an agreed definition, denominator, data source, owner and interpretation. In some cases, the definition will require input from researchers, data specialists, governance teams and affected communities.
Putting a number around a judgement does not suddenly make it objective. We need to be open about who made the judgement, what it was based on and what the resulting measure can actually tell us.
Not every measure needs a target
Some measures are there simply to keep us honest. They show whether progress in one area is creating a problem elsewhere.
Turning privacy incidents, for example, into a simplistic numerical target can create the wrong incentive if it discourages reporting. Representation may need to be understood in relation to a specific research purpose rather than pursued as a single universal percentage.
The role of an OKR is to focus effort on a meaningful change, not to absorb every responsibility of the product team.
The principle has not changed.
Ecommerce and research data products sit in very different environments. One has a visible purchase at the end of the journey. The other may contribute to social value through a chain that takes years to unfold.
But the measurement discipline is the same.
Do not confuse activity with value. Do not turn every available metric into a KPI. Do not turn every KPI into a key result. Be clear about what the product controls, what it influences and what it can only help make possible.
The purpose of measurement is not to prove that the product caused everything that happened afterwards.
It is to show, honestly and with evidence, how the work contributed to the outcome the organisation exists to create.
Leave a Reply