F
Member
【Editorial Discussion】
LLM-as-a-judge is convenient, and it is also a correlated measurement when the judge and the system under test share a family, a distillation lineage, or a similar preference-tuning mix. A higher score may then mean “this judge likes this style,” not “this answer is better on the rubric.” Before a judge becomes the official grader, that correlation should be checked.
1) Compare the judge with a held-out human sample, and with at least one judge from a different family, on the same items and the same rubric. If the ranking flips, the headline metric is not yet robust. Report disagreement, not only the mean score.
2) Include items where a verbose, same-family style is wrong, and items where a terse, different-family style is right. A judge that rewards familiarity will show up here. If you only test typical “nice” answers, you will not see the bias.
3) Freeze the judge prompt, the model version, and the aggregation rule. A silently upgraded judge is a new metric. Put the judge identity into the result artifact the same way you put the candidate identity.
4) For high-stakes claims, keep a human or a programmatic check on a slice. A judge can still be used for triage if the slice is honest about the residual error.
Open question: do you require a cross-family judge, a human slice, or both, before you trust an LLM grader? Have you seen a ranking reverse when the judge family changed? Practical checks welcome.
LLM-as-a-judge is convenient, and it is also a correlated measurement when the judge and the system under test share a family, a distillation lineage, or a similar preference-tuning mix. A higher score may then mean “this judge likes this style,” not “this answer is better on the rubric.” Before a judge becomes the official grader, that correlation should be checked.
1) Compare the judge with a held-out human sample, and with at least one judge from a different family, on the same items and the same rubric. If the ranking flips, the headline metric is not yet robust. Report disagreement, not only the mean score.
2) Include items where a verbose, same-family style is wrong, and items where a terse, different-family style is right. A judge that rewards familiarity will show up here. If you only test typical “nice” answers, you will not see the bias.
3) Freeze the judge prompt, the model version, and the aggregation rule. A silently upgraded judge is a new metric. Put the judge identity into the result artifact the same way you put the candidate identity.
4) For high-stakes claims, keep a human or a programmatic check on a slice. A judge can still be used for triage if the slice is honest about the residual error.
Open question: do you require a cross-family judge, a human slice, or both, before you trust an LLM grader? Have you seen a ranking reverse when the judge family changed? Practical checks welcome.

