PairwiseComparisonEvaluator: with enforce_consensus=True, a tie in the original order plus a decisive flipped order returns the flipped judge's score/passing unconverted (verdict inverted)

[shaurya416]

llama-index-core/llama_index/core/evaluation/pairwise.py:206-218 (HEAD 2232f3019f7aa581f455583262100111dff4a4d3)

        # get the judges (original and flipped) who voted for answer_1
        voters_1 = [eval_result] * (eval_result.score == 1.0) + [
            flipped_eval_result
        ] * (flipped_eval_result.score == 0.0)

        # get the judges (original and flipped) who voted for answer_2
        voters_2 = [eval_result] * (eval_result.score == 0.0) + [
            flipped_eval_result
        ] * (flipped_eval_result.score == 1.0)

        if votes_1 > votes_2:
            return voters_1[0]  # return any voter for answer_1
        elif votes_2 > votes_1:
            return voters_2[0]  # return any vote for answer_2

The class contract (pairwise.py:99) is "Outputs whether the response given is better than the reference response", so score/passing are expressed in the original frame: 1.0/True means response beat second_response.

flipped_eval_result comes from a call with the answers swapped (_get_eval_result(query, second_response, response, reference), :263-265), so its score and passing are in the flipped frame: 1.0 means second_response won. The vote tally at :200-201 accounts for that (1 - flipped_eval_result.score), but the return at :216/:218 hands back the flipped EvaluationResult unconverted.

voters_1[0]/voters_2[0] is the original result whenever the original judge was decisive, because it is listed first. When the original judge answers [[C]] (score 0.5) and the flipped judge is decisive, the original result is in neither voter list, so the flipped result is returned as-is.

The path needs only enforce_consensus=True (the default), a tie in the original order and a decisive reply in the flipped order.

From reading the function: pairwise_source=FLIPPED is returned only in these two cases. If the original judge is decisive and the flipped judge agrees or ties, the original result is returned; if they disagree, the result is NEITHER. So every result currently carrying pairwise_source=FLIPPED has an inverted score and passing.

Measured

Reproduced at HEAD 2232f3019f7aa581f455583262100111dff4a4d3 (pairwise.py sha256 prefix 61301043a36027e6) by running the unmodified pairwise.py through importlib, without installing the package. Substitutions used:

• Stubbed imports: prompts, LLM, Settings, the Response schema, and a pydantic bridge pointing at pydantic 1.10.
• EvaluationResult: the real evaluation/base.py model in one run, and a dataclass with the same fields in another. Both runs gave identical results.
• Judge: a scripted stand-in that returns fixed replies and records the answer_1/answer_2 it was shown in each call.

Expected values come from decoding each reply to the answer it actually picked, not from the score arithmetic.

original reply	flipped reply	who actually won	returned (passing, score, source)	expected (passing, score)
[[A]]	[[B]]	response 2.0/2	True, 1.0, ORIGINAL	True, 1.0
[[B]]	[[A]]	second_response	False, 0.0, ORIGINAL	False, 0.0
[[A]]	[[C]]	response 1.5/2	True, 1.0, ORIGINAL	True, 1.0
[[B]]	[[C]]	second_response	False, 0.0, ORIGINAL	False, 0.0
[[C]]	[[C]]	tie	None, 0.5, ORIGINAL	None, 0.5
[[A]]	[[A]]	position bias	None, 0.5, NEITHER	None, 0.5
[[B]]	[[B]]	position bias	None, 0.5, NEITHER	None, 0.5
[[C]]	[[B]]	response 1.5/2	False, 0.0, FLIPPED	True, 1.0
[[C]]	[[A]]	second_response (response 0.5/2)	True, 1.0, FLIPPED	False, 0.0

Controls:

• The seven non-bold rows return the expected verdict.
• With enforce_consensus=False, the three single replies return (True, 1.0), (False, 0.0) and (None, 0.5) as expected, so the defect is in the resolve step.

On repetition, ten rows of [[C]]/[[B]] (response wins every row) report mean score 0.0 and passing=False on 10/10. Ten rows of [[C]]/[[A]] (response loses every row) report mean score 1.0 and passing=True on 10/10.

Only pairwise_source == FLIPPED marks the affected rows. There is no test module for pairwise.py under tests/evaluation/.

Consequence

For the affected rows, consumers that read .score or .passing get the opposite verdict, and nothing in those fields says so. Any win rate computed over score or passing counts these rows the wrong way.

Based on reading the cells of docs/examples/finetuning/llm_judge/pairwise/finetune_llm_judge.ipynb, without re-running them:

• Cells 28 and 44 store final_eval_result.score per judge.
• Cell 49 counts both "original" and "flipped" rows as conclusive.
• Cells 49 and 51 compare the raw scores across judges.

For agreement rate, a FLIPPED row counts with the wrong sign. Correlation and Jaccard are skewed in the same direction, but by an amount that depends on how many FLIPPED rows fall inside the mask. That amount was not measured.

The notebook also uses pairwise_source to pick which chat history to keep for fine-tuning. That selection is consistent within the flipped frame and is not affected.

Suggested fix

The same function already converts the flipped judge to the original frame when it counts votes (votes_1 = eval_result.score + (1 - flipped_eval_result.score), :200-201). Apply the same conversion when a flipped result is returned:

• Keep pairwise_source=FLIPPED and the judge text.
• Return score = 1.0 - flipped_eval_result.score and passing = (flipped_eval_result.score == 0.0).

A flipped voter is selected only when its score is exactly 0.0 or 1.0, so passing is never None on that path. The NEITHER branch at :227-234 already builds a fresh result in the original frame and can serve as the model. A parametrized regression test over the nine (original, flipped) reply pairs above, asserting the original-frame verdict, covers the change.

This does not overlap open PR #22599, which changes only verdict-token parsing in _default_parser_function.

Happy to open the PR.