The strongest case against preference models

The strongest objection to the paper is that a transparent score can still give an unsuitable decision process the appearance of discipline. Visibility does not establish that the criteria are legitimate, that the ratings have meaningful intervals or that the people choosing weights have authority. A reproducible calculation can faithfully reproduce an unjustified architecture. The paper's thesis therefore has a condition: transparency and sensitivity improve the defensibility of a preference comparison only when its decision envelope, evidence and operations fit the action being requested.

The first challenge concerns comparability. Impact, readiness and risk ratings can look cardinal because they are numerical, while their underlying rubrics may support only broad ordinal statements. Linear normalisation and weighted addition then create precision that the evidence does not warrant. A difference between two ratings is treated as a proportional preference movement within the range. If the team cannot defend that representation, exposing the formula does not cure the problem. The appropriate repair may be a better rubric, a qualitative comparison or another method.

The second challenge concerns legitimacy. Some values should not be traded away through compensation, and some participants may have rights or decision roles that an average vector cannot represent. A clean contribution ledger can show exactly how a binding concern was outweighed without making that outcome legitimate. Eligibility and authority must therefore be established outside the sum where required. The score should operate within those conditions. It should not be used to redefine them merely because their separate treatment makes the process less convenient.

The third challenge concerns constructed universes. The model compares the candidates supplied, rather than discovering every relevant option. Sponsors and analysts can shape the universe through scope, inclusion and timing. A high-ranked proposal may lead only because a credible alternative was never developed. The ex-ante option register reduces that discretion but does not prove completeness. The recommendation should state the search and inclusion perimeter. A ranking over a limited universe is not a global claim about what deserves capital across the organisation.

The fourth challenge concerns meaning across scales. Observed-range normalisation can move totals when a candidate is added or a competitor's rating changes, even while an option's own raw performance stays fixed. This makes the score difficult to interpret as a stable measure of absolute value. The paper responds by treating it as a conditional preference within a frozen set and by showing the range artefact explicitly. That response preserves a useful comparison role, but it concedes that cross-set score comparisons are generally inappropriate without a different scale architecture.

The fifth challenge concerns institutional use. A model can be technically transparent while readers see only the final ordering, a sponsor cites it as independent endorsement or a committee treats it as a substitute for judgement. Documentation cannot guarantee appropriate use. The process needs a use statement, a conditional recommendation and a record of the actual action. The paper's contribution is an architecture for that connection, not a claim that a numerical tool will automatically improve organisational behaviour.

The sixth challenge concerns the seductive simplicity of a single leader. Many strategic decisions involve incompatible objectives, different stages and feasible combinations. A single ordering can compress these differences into a format that feels decisive. The committee may then optimise the display rather than resolve the actual problem. The paper responds by distinguishing preference, eligibility, funding, learning and authority. If those distinctions make a single leader less useful, the method should yield to a different representation instead of defending the convenience of its output.

There is also a serious case for expert judgement without numerical aggregation. Experienced decision makers can recognise scope defects, operating dependencies and implausible narratives that a fixed model does not contain. For some decisions, a structured deliberation with explicit reasons may be more informative than ratings. The objection is not that judgement is error-free. It is that forcing expert interpretation into a shallow numerical architecture can discard the knowledge that matters. A transparent qualitative record can retain that knowledge while remaining challengeable.

The counterargument is that unstructured judgement also hides assumptions and can be inconsistent across proposals. A numerical model can make tradeoffs inspectable, reveal double counting and expose conditional reversals. Its value lies in those operations when their meaning is adequate, rather than in claiming objectivity. The paper does not choose arithmetic over judgement as a universal rule. It argues for an accountable representation of judgement and evidence, using a weighted model when that representation fits and recognising alternatives when it does not.

Sensitivity is vulnerable to its own performance of rigour. A report can include many scenarios while avoiding the consequential uncertainty. The selected shifts may be arbitrary, the raw ranges may be implausible or the governing mandate may remain unresolved. The paper responds by asking each challenge to identify what changes, why the change is relevant and what action follows. A large sensitivity appendix is not itself evidence of robustness. The challenge plan needs a decision perimeter and a falsifying consequence.

The app's raw Pareto check is useful but bounded. It can flag an option that lacks a represented advantage, while omitting unmodelled features and relying on the supplied evidence. A dominated result does not prove universal inferiority. A non-dominated result does not establish affordability, permission or strategic desirability. The paper uses the check as an adversarial question rather than a certification. This avoids overstating a mathematical property of the representation as a complete conclusion about the proposal.

Use the frozen studies to inspect the engine's mandate reversals, tied scores, ignored-criterion dominance, range movement and bounded evidence changes under authored inputs. They establish mechanisms and counterexamples within that representation. To test whether the workflow improves investments, define an empirical study of actual decisions and outcomes with attention to selection and context. No comparative committee field trial is reported here. The methodological defence remains conditional until that separate evidence exists; numerical demonstrations cannot substitute for it.

Keep workflow evaluation at the level its evidence supports. Reproduction establishes a supplied preference-model behaviour; a clearer meeting record establishes a process property. Better investment outcomes require a further credible comparison with the decisions that would otherwise have been made and an account of changing conditions. One property does not establish the next. A practitioner can examine whether the raw ledger locates a dispute sooner or a recorded condition changes an action, while keeping those bounded observations separate from returns or institutional-misconduct claims. The evaluation design should identify the proposed effect before selecting its observations.

The practical test is whether the representation improves a particular decision. Can participants explain the criterion meanings, understand the tradeoffs, challenge the evidence and connect the result to an authorised action? Can a reviewer reproduce the numerical study and identify its limits? Can a changed assumption lead to a controlled revision rather than retrospective storytelling? If the answer is no, adding more decimal places or more polished charts does not fulfil the paper's thesis. The process needs repair or replacement.

Remove the weighted total as an adversarial test of the architecture. If the team can still identify the admissible alternatives, the decisive evidence gap and the next useful action, those components carry independent decision value. If the recommendation collapses because nobody can explain the leading score, the total has displaced the argument. A package problem may retain raw criteria and dominance but require another selection representation; a value dispute may retain stakeholder scenarios while withholding a collective vector. This subtraction test identifies which component is doing useful work. It supports selective adoption rather than making the complete scoring workflow an institutional prerequisite.

The strongest conclusion is therefore bounded. Transparent preference modelling and relevant sensitivity can make a compensatory standalone comparison more defensible than an opaque ranking, provided the evidence, scales, universe and mandate are adequate to the present action. When non-compensable conditions, package interactions or contested authority govern the choice, those elements require their own process. The methodology succeeds by making that boundary clear, even when the appropriate recommendation is to stop using the additive app for the requested commitment.