Expert elicitation needs discipline

Expert judgement can be necessary when relevant data are incomplete, sparse or poorly transferable. The institution should not disguise that judgement as an observed frequency. It should state the question posed, the information available to the experts, the elicitation process and the way the result enters the decision. A structured judgement can be valuable because its reasoning and uncertainty are reviewable. Its authority does not come from the apparent precision of a consensus number or the prestige of the people who supplied it.

The elicitation question should use the same event definition as the model. Asking whether a project is promising can produce a judgement about attractiveness, feasibility or preference, while the app requires a probability of a defined binary state. Experts may answer different propositions without realising it. A facilitator should therefore test the wording and document the horizon, population, conditions and exclusions. Clarifying the event can be more important than choosing an aggregation technique for numbers that initially meant different things.

Experts should also distinguish their belief about the event from their confidence in the supporting evidence. A specialist may offer a probability while emphasising that the estimate depends heavily on an unresolved operating condition. Another may feel confident about the process but assign a less favourable likelihood. The register should retain both the event judgement and its evidence limitations. It should not convert a confidence adjective into an unsupported probability or use a narrow display range to imply that disagreement has disappeared.

Selection of experts should follow the knowledge needed for the event. Technical, commercial, operational and institutional uncertainties may require different experience. A large panel can still omit the knowledge of the people who implement the response or bear a consequence. The paper does not prescribe a universal panel size or invent a real panel. It recommends a knowledge coverage record, disclosure of relevant interests and a review of important missing perspectives. The commissioning authority should understand what the panel is equipped to judge and what remains outside its competence.

Information sharing can improve understanding but also reduce the independence of judgements. Experts may begin from the same report, adopt a dominant participant's framing or align after a discussion. The elicitation record should preserve what was shared and when, especially if the final outputs will be treated as separate evidence contributions. A consensus can be useful for a committee decision, but its existence is different from independent confirmation of the state. The dependency chapter's distinction applies equally to a panel and to a stack of published reports.

The facilitator should make reasons for judgement visible. A point estimate without an explanation gives reviewers little ability to assess transfer or identify what new evidence could change it. Experts can describe mechanisms, critical assumptions, conflicting observations and conditional scenarios. These explanations can improve the action architecture even when the numerical range remains broad. The purpose is to capture decision-relevant knowledge rather than extract a convenient percentage that the optimiser can process without further discussion.

The process should reduce avoidable distortions without claiming to eliminate judgement. Participants may be influenced by an initial number, a desired project outcome or incentives associated with a recommendation. A structured protocol can separate initial estimates, discussion and revision, and can require explicit reasons for changes. The institution should record the protocol and any departures. The paper does not claim that a particular protocol guarantees calibrated probabilities. Its practical value is to make the conditions of judgement sufficiently transparent for an independent reviewer to challenge.

Aggregation is a modelling choice. Averaging individual estimates can be simple and reproducible, but it does not automatically resolve different event definitions, expertise, dependencies or interests. Another aggregation rule may be appropriate in a defined context, provided its rationale is recorded. The binary app does not combine panel judgements or infer a distribution from them. A real elicitation output should be processed through the declared method before it becomes a supplied prior or likelihood, and the transformation should be retained with the evidence.

Ranges should have a meaning. A lowest and highest estimate across experts is different from a probability interval supplied by one expert, and both differ from an interval estimated from data. The committee should not treat them interchangeably. If the elicitation is used only to select plausible sensitivity fixtures, it should say so. If it supports a probabilistic model of parameter uncertainty, that model requires an explicit method. The paper's contrasting studies are authored scenarios, not a panel distribution or an empirical confidence interval.

Calibration and performance evidence require a relevant record. A specialist's experience can justify including them in a panel without proving the numerical accuracy of every judgement. If a commission relies on a claim of expert calibration, it should identify the questions, outcomes, scoring method and transfer conditions that support it. The paper makes no such claim for hypothetical role holders. It keeps the elicitation record at the level it can honestly support: a structured process for documenting judgement and identifying the evidence needed to assess its numerical use.

The institution should preserve disagreement that changes the decision. A report that publishes only a consensus prior can conceal that credible alternatives select different actions. The app can compare explicitly supplied fixtures to show that sensitivity, while the source register retains the reasons for each alternative. The committee can then decide whether to commission evidence, choose a more robust action or accept a documented exposure. Suppressing consequential disagreement to make a decision look settled defeats the purpose of eliciting informed judgement.

Judgement may need updating when new evidence arrives. The update should have a stated trigger and a preserved history. An expert's revised number can reflect a new observation, a corrected event definition or a change in incentives and interpretation. The reasons matter. A new number should not silently overwrite the original basis of the research purchase. The institution should retain enough context to determine whether the current prior actually represents the information set it claims to represent.

Disagreement can be diagnosed before averaging expert numbers. Hypothetical experts may be answering different success events, using different information sets or applying different assumptions about implementation. Resolve those differences into a common question or retain separate scenario propositions. If they share the question but disagree about likelihood, preserve the reasons and test the decision under the distinct judgements before selecting an aggregation method. If every defensible judgement produces the same available action, further convergence may have little immediate decision use. If the policy changes, the disagreement itself becomes part of the commission's sensitivity. This routes elicitation effort to a consequential distinction rather than make consensus a production target.

A protocol should retain the judgement that discussion displaced. In a hypothetical panel, an expert revises a view after learning of a previously missing observation. Record that observation and the changed rationale. A revision after hearing the chair's preferred action has a different interpretation and should not be described as new empirical support. The retained individual and revised accounts allow the facilitator to distinguish learning, clarified definitions and social convergence. They do not establish calibration without a relevant performance record. This makes the elicitation output inspectable and keeps a formatted panel conclusion from acquiring the status of an observed frequency merely because participants finally agree.