Skip to content
ICM Holdings International
Search⌘ K
Index
Choosing what deserves capital

18 · Governance and application

Performance review must preserve evaluation logic

Compare actual outcomes against original objectives and reconstruct why rankings changed

148 pages · Edition 2026-10-10

Governance and application

A portfolio learning and change-control record

Keep the original logic while responding to material change

A decision model has a controlled life after the meeting. New evidence, changed objectives or altered constraints can make its recommendation less applicable. The organisation should decide in advance which changes require review and who owns that review. Without triggers, the model may be reused long after its scope has changed. With indiscriminate review, every minor update can reopen the whole debate. A useful process focuses on changes that can alter eligibility, preference, feasible action or the conditions of the adopted commitment.

The original logic should remain available. Archive the charter, inputs, governing vector, model version, conditional recommendation and actual action adopted. The archive is not merely a collection of output screenshots. It needs the assumptions and reasons that explain the action. A later reviewer should be able to distinguish a calculation error from an unsupported estimate, a changed mandate or a failure to satisfy a condition. Those distinctions guide repair. Recalculating the current numbers alone cannot reconstruct what the earlier committee knew and authorised.

The review should first ask whether the decision is still the same. A change from investigating an option to implementing a programme can require a different evidence standard and method. A new funding envelope can make package selection relevant. A changed service objective can alter the candidate universe. If the envelope has changed, the organisation should revise the charter rather than simply update values in the old model. Numerical continuity does not guarantee continuity of the decision being made.

Performance evidence should be updated proposition by proposition. If a verified operating plan resolves part of readiness, record the corresponding interpretation and remaining uncertainty. If a cost estimate includes a newly recognised enabler, identify the scope change before comparing it with other candidates. The team should avoid treating every revised value as comparable with the old one. A changed perimeter may require reassessment of the whole set. The review record should say whether the change concerns evidence within a fixed definition or a revision to the definition itself.

Evidence and scope

Assumptions, alternative evaluation and later verification should be explicit.

Evidence: Circular A-94: Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs

A completed review-trigger register

The original fictional register ties each trigger to a specific review rather than reopening the model for every event. No event is claimed to have occurred.

Scroll for all columns. With the table focused, use the left and right arrow keys.

A completed review-trigger register
TriggerWhat may changeReview ownerRequired response
Operating design verified or failsReadiness interpretation and an implementation conditionOperating owner and committee analystUpdate the supported proposition, recompute the scenario and apply the closure disposition
Cost perimeter expandsRaw cost comparability and funding feasibilityCost-estimate ownerAlign the perimeter across candidates before treating score changes as performance changes
Strategic envelope revisedObjective, weights and candidate universeAuthorised committeeReissue the charter and adopt a mandate suitable for the new decision
Enabler becomes unavailableEligibility, scope or dependency feasibilityProgramme ownerReconsider the affected action and reserve options outside standalone ranking
New credible candidate admittedObserved ranges and comparison setOption-register ownerApply the inclusion rule and recompute the entire coherent set
Model version changesComputational semantics or validationModel maintainer and analystReproduce archived fixtures and disclose any changed result before using the new version

Original change-control exhibit. The response concerns the decision architecture and evidence record, not an automated action by the app.

A retrospective review should compare propositions with the evidence later obtained, without converting the old score into a forecast it never was. The model did not predict a financial return or a probability of success. It expressed preference under supplied ratings. A useful review asks whether the readiness interpretation was supportable, whether the claimed impact mechanism operated and whether the scope omitted material commitments. Those findings can improve rubrics and evidence standards. They should not be described as calibration of a probability that the model did not estimate.

Outcome review needs attention to counterfactual limits. An implemented proposal's observed outcome may depend on context, execution choices and other interventions. A favourable outcome does not prove that the original selection was uniquely optimal, and an adverse outcome does not automatically prove that the preference model was wrong. The organisation should evaluate the relevant mechanism with an appropriate method and distinguish selection quality from delivery performance. The archive helps identify what was assumed and what actually changed after approval.

Process review asks whether the decision was made as intended. Were eligibility conditions resolved? Was the governing mandate authorised? Did the sponsor implement the scope that was scored? Were review triggers acted on? These questions can be answered without claiming a causal estimate of programme impact. A process failure may explain why a plausible recommendation did not translate into the intended action. Repair then concerns governance or execution rather than recalibrating the weights to fit the later outcome.

The team should avoid hindsight rewriting. Once a result is known, it is easy to describe the winning or failed proposal as obviously superior or inferior. Preserve the evidence available at the decision date and the uncertainty that remained. A fair review asks whether the action was defensible under that information and authority, while still learning from later findings. This does not excuse avoidable mistakes. It prevents the organisation from confusing bad luck, new context and weak original reasoning.

A changed weight vector requires a changed rationale. The committee may legitimately value readiness more after a service obligation becomes urgent, or impact more after adopting a strategic expansion envelope. The record should identify the event and authority behind that revision. It should not describe a vector adjusted to match a realised outcome as evidence that the new weights were always correct. Preferences are governed judgements, not parameters to be retrospectively fitted without a decision mandate.

Model maintenance also needs a distinction between corrections and enhancements. A correction may repair a validation error or a numerical defect while preserving intended semantics. An enhancement may change normalisation, sensitivity or the criteria represented. Archived scenarios should be recomputed and compared under the new version, with any changed interpretation disclosed. The platform's version validation prevents importing an unsupported version silently. The organisation still needs a release record explaining why the change was made and which earlier decisions require reconsideration.

A new candidate is not merely another row. Under current-set normalisation it can change every transformed range. The review should therefore generate a new scenario rather than append the candidate to a briefing containing old scores. The original set remains a valid record of the earlier comparison. The new set answers an updated question. Keeping both with their dates, reasons and scope prevents a reviewer from mistaking scale movement for a change in individual performance.

Monitoring can reveal that the score selected a proxy rather than the intended outcome. A programme may increase the number of functioning sensors while leaving operational decisions unchanged because no team uses the readings. The installation count then supports delivery, while the benefit mechanism remains unestablished. Trace the chain from installed capability to use and from use to the valued service before revising impact. This diagnostic distinguishes a failed intervention, a failed adoption route and a failed indicator. Assigning a benefits owner is useful only when that owner can investigate those links rather than report the convenient quantity repeatedly.

A changed ranking after commitment creates a switching question with a different reference action. Continuing the installed intervention may avoid transition disruption, while switching may require new integration, stranded work or retraining. The original acquisition comparison does not contain those consequences. First identify what can still be changed and which costs or obligations belong to each current route; then select a method capable of comparing continuation, adjustment and replacement. A reserve option's earlier preference score is relevant history, not a sufficient switching justification. Continuing despite a lower present score can be rational when the new alternatives face consequences absent from the old choice.

Select review events by the causal position of the changed information. A verified cost exclusion can change the commitment perimeter immediately; a new candidate can change range endpoints even without altering existing raw values; a minor operating observation may require a longer comparable record before changing an impact judgement. These events should not all trigger the same mechanical refresh. Identify the affected proposition and dependent comparison first. An unchanged mandate with stable evidence needs a status check, while a changed universe needs scale interpretation and a changed objective needs a new mandate. This triage reduces repeated calculation that adds no decision information.

Historical reconstruction should preserve two views of a changed proposal. The contemporaneous view asks whether the original action followed reasonably from the evidence and preferences available then. The present view asks what action is justified now. Using today's evidence to overwrite the earlier fixture destroys the first view; continuing to apply the earlier fixture as current advice destroys the second. Keep the editions and connect them with a proposition-level difference explanation. Exact input and output records support reconstruction, while the explanation identifies whether the change concerns performance, preference, eligibility or the engine. Neither view alone supplies the whole learning account.

Review the proposition and the action, not only the score

Preserve the original charter, evidence, vector, engine version and adopted action. Apply predeclared triggers to determine whether the present decision remains within that envelope. Classify the change as evidence, definition, mandate, feasibility or computation. Update a coherent scenario and explain its consequence for the action. Review outcomes with an appropriate evaluation method and processes against their conditions. Retain the earlier edition so hindsight cannot rewrite the original basis. Close the review with a disposition, owner and next material trigger.

The case for a continuously refreshed ranking

A frequently updated ranking can make new evidence visible and support an active development pipeline. It is useful when changes have defined owners and consistent scope. Its weakness is that automatic refresh can obscure changed ranges, mandates or commitment stages while making the result look like a continuous measure. The repair is to version consequential changes and show the reason for each new scenario. Refresh should improve applicability, rather than create the appearance of new analysis without a changed proposition.

The model does not monitor outcomes or enforce reviews

Use accountable institutional roles and appropriate evaluation methods to gather operating evidence, assess causal outcomes, enforce review triggers and verify conditions. Portfolio Compass only recomputes the supplied scenarios; the fictional review register assigns those duties without performing them. Keep the retrospective score within its original meaning as conditional preference. It was never a probability or monetary prediction, so later review cannot treat it as a calibrated forecast. Reconsider the action through the relevant evidence and mandate when either has changed.

Evidence at the point of use

Open a citation in the chapter to inspect its source, stated support and limits.

Find the next connection

Open the full search desk