Governance and application
A staged-investment decision protocol
Make learning milestones capable of changing a later action
A pilot is useful when a bounded commitment can resolve a proposition that matters to a later action. It should have a question, a credible observation method and a decision that remains open after the result. A small implementation labelled pilot does not automatically create learning. If the organisation has already committed to scale regardless of the findings, the activity may be a rollout stage rather than a choice experiment. That purpose can still be legitimate, but the recommendation should describe it accurately.
The learning question should be narrower than the ambition of the whole programme. A pilot may test whether the operating team can respond to an alert, whether an interface supports a specified workflow or whether users adopt a process under stated conditions. It cannot necessarily establish the full long-term impact of a scaled intervention. The team should identify which part of the outcome mechanism the bounded activity examines and which parts remain outside it. This makes a favourable result informative without exaggerating what it proves.
The committee should connect the question to the disputed rating or condition. If readiness is uncertain because no response workflow has been demonstrated, the pilot should examine that capability. If impact is uncertain because the causal mechanism is unclear, a process demonstration may not resolve it. A pilot designed around easy-to-measure activity can leave the consequential uncertainty untouched. The evidence record should state which input interpretation or commitment condition would change under different findings.
A useful pilot design includes an alternative explanation challenge. Improvement during the activity might reflect seasonal conditions, additional temporary staff or changes occurring elsewhere rather than the proposed intervention. The appropriate evaluation design depends on the question and context. The team should establish what comparison or process evidence is feasible and what causal conclusion it can support. The paired preference app does not perform evaluation analysis; it receives the resulting performance interpretation only after the relevant evidence process has been reviewed.
Evidence and scope
Evaluation planning should inform intervention design and subsequent learning.
Evidence: Magenta Book: Central Government guidance on evaluation
A completed fictional learning protocol
The original protocol examines whether an operating response capability supports a later readiness interpretation. It contains no claimed measured effect or predetermined scale decision.
Scroll for all columns. With the table focused, use the left and right arrow keys.
| A completed fictional learning protocol | ||
|---|---|---|
| Protocol element | Completed design statement | Decision consequence |
| Question | Can the accountable team receive, interpret and act on a defined operational signal within the intended workflow? | A favourable result can support that capability component of readiness; it does not establish programme impact |
| Scope | A bounded workflow demonstration using the proposed roles and interfaces | The finding applies to the demonstrated configuration and conditions |
| Alternative explanations | Record temporary support, concurrent changes and departures from normal operation | Do not transfer a favourable finding to unsupported routine conditions |
| Adverse finding | Identify whether the failure concerns data, interface, training or operating responsibility | Redesign the failed component and reconsider the readiness interpretation |
| Scale decision | Return to the committee with the capability evidence and remaining programme requirements | Scaling remains a separate authorised commitment |
| Exit | Stop the bounded activity when its question is answered or the protocol cannot produce useful evidence | Avoid indefinite operation under the label of learning |
Original completed learning design. The committee must define an appropriate evaluation method for its actual operating context.
A milestone should describe a finding or verified capability, rather than merely an activity completed. A report delivered, a workshop held or a system installed can be necessary work without resolving the learning question. The committee should ask what conclusion the milestone permits and what remains uncertain. This prevents a staged plan from progressing automatically through administrative completion while the relevant decision proposition remains unsupported. Progress should be linked to evidence that changes the readiness of the next commitment.
The protocol should specify favourable, adverse and inconclusive findings. An inconclusive result is not a disguised success. It may mean the activity lacked sufficient variation, the measurement failed or the context did not expose the condition of interest. The team should decide whether redesign, a different method or stopping is appropriate. Predeclaring this disposition reduces the temptation to interpret any activity as evidence for continuation. Learning includes discovering that the proposed experiment cannot answer the question.
A credible stopping rule protects the option to learn. It identifies conditions under which the organisation will stop, redesign or decline scaling and names who can make that decision. The sponsor should understand the rule before the pilot begins. If cancellation is institutionally impossible or the initial commitment creates unavoidable continuation obligations, the staging claim needs qualification. The organisation may still proceed, but should recognise the exposure it has accepted rather than treating the next stage as freely optional.
The pilot should use a scope representative enough for the proposition being tested. A demonstration with exceptional staff, simplified interfaces or unusually supportive conditions may reveal technical possibility while saying little about routine capability. That can be useful if the question is technical feasibility. It cannot automatically support readiness for normal operations. The record should distinguish the environment tested from the environment of the proposed implementation. A transfer plan needs evidence about the differences that matter.
The cost of learning belongs in the decision. A study or pilot consumes resources, delays some commitments and may create temporary operating burden. Its value depends on whether the information can improve a later choice enough to justify those costs under the relevant decision framework. This paper does not calculate expected information value for the portfolio fixtures. It asks the team to identify the decision that learning changes, the plausible findings and the burden of obtaining them. A vague promise to learn is insufficient to justify an indefinite programme.
Information can also arrive too late to matter. If procurement or implementation becomes irreversible before the pilot result is available, the activity may not inform the claimed decision. The team should align the learning milestone with the commitment sequence. A useful schedule shows when the choice remains open and what resources must be reserved to preserve it. The pilot recommendation should avoid claiming a staged option when the organisation has already crossed the consequential commitment boundary.
The ranking can help choose what to investigate when its decision envelope is designed for that purpose. A development allocation may emphasise the importance of resolving an uncertainty and the feasibility of learning, rather than the same readiness rubric used for implementation. The canonical app has fixed criteria and does not calculate an optimal learning programme. The team should therefore avoid pretending that an implementation ranking automatically selects pilots. It can author a clearly bounded preference scenario if the existing attributes are meaningful, or use a more suitable learning method.
A favourable pilot result should update a proposition, not simply boost every rating. Demonstrating a response workflow may support readiness while leaving strategic impact and execution risk partly unresolved. The analyst should identify the specific rating interpretation changed and retain the other assumptions unless new evidence justifies revision. This discipline makes the recomputed preference informative. A general uplift after a successful activity can double-count optimism and make it impossible to understand which evidence drove the new recommendation.
An adverse result can improve the decision even if it lowers the proposal's score. It may prevent a larger unsuitable commitment, identify a necessary redesign or clarify that an alternative is preferable. The sponsor should not be rewarded only for positive findings. A credible process values information that changes the action appropriately. The committee can preserve the pilot record as evidence of disciplined learning rather than treating a stopped proposal as a failure of analysis or effort.
At pilot close, compare the finding with the adverse explanation specified before the activity. A smooth demonstration may show that the process works with intensive specialist assistance while leaving ordinary support unresolved. Record success for the assisted condition and keep the ordinary-support branch open. If the protocol expected unsupported operation, that result fails the intended test even though the visible output is excellent. If assistance was part of the intended operating design, its continuing availability and commitment belong in the next scope. This comparison makes the pilot informative without rewriting its success criterion to match the outcome.
Transfer from a pilot by inspecting the conditions that produced its result. A small trial may exclude peak duty, difficult interfaces or the proposed operator population. The next commitment should target whichever omitted condition could defeat the mechanism, rather than repeat the same demonstration at greater expense. If the omission is an integration interface, test that interface; if it is sustained duty, investigate performance over the relevant duty; if it is operator capability, examine ordinary staffing. A successful trial can justify these next tests while remaining insufficient for full implementation. The staging sequence should progressively remove distinct failure explanations.
Approve a learning commitment with an open later choice
Name the proposition and later action that the pilot can change. Define a bounded scope and an evaluation approach capable of examining the proposition. Predeclare favourable, adverse and inconclusive dispositions, along with a credible exit and responsible decision maker. Align the finding with the commitment sequence so the later choice remains open. Update only the input interpretations supported by the findings, recompute the relevant scenario and retain the transfer limits. Authorise the next stage on the evidence obtained rather than on activity completion alone.
The case for implementing and learning simultaneously
Some interventions can be improved during implementation, and a separate pilot may delay useful service. That approach can be reasonable when the commitment is sufficiently reversible, the learning question remains observable and the organisation can change course. Its weakness is calling an effectively irreversible rollout a learning option. The repair is to identify the commitment boundary, residual exposure and decisions that can still change. Where a bounded experiment can resolve a consequential uncertainty before that boundary, a separate stage may be justified.
No empirical effect or information value is estimated
The completed protocol is fictional and contains no measured pilot result. Portfolio Compass does not design experiments, estimate causal effects, calculate sample sizes or value information. A rating update requires an appropriate evidence process outside the app. The chapter supplies a decision architecture for learning, with scope and stopping conditions. It cannot certify that a proposed pilot will resolve the uncertainty or that its information benefits exceed its costs.