Position-bias harness for local LLM judges
Some LLM judges prefer a slot, not an answer. dark-current-mlx replays every pairwise comparison in both orders and measures how often the verdict follows the position instead of the content.
Open-weight judges on a MacBook via Apple MLX. Greedy decoding, fixed seed, one auditable JSONL record per call. No cloud, no paid APIs.
Pairwise evaluation trusts a judge model to pick the better of two answers. Many judges also care about where an answer sits. Swap A and B, and the verdict can follow the slot rather than the content.
In a photodetector, dark current is the signal you measure with no light coming in. In a judge, it is the preference you measure when no real quality difference is left. That leftover signal is what this harness isolates, per judge and per condition.
Measuring this usually drifts toward paid API baselines or GPU pipelines. Here the whole first pass stays on a laptop: local, deterministic, and cheap enough to rerun after every change.
The metric is strict on purpose. A judge gets credit for consistency only when its verdicts survive the swap in both directions.
Every pairwise item is judged in original order (A/B) and in swapped order (B/A).
Only exact [[A]], [[B]] or [[Tie]] markers count. Prose that skips the marker is recorded as unknown, never guessed.
An item is consistent only if the judge flips A to B, flips B to A, and keeps Tie as Tie. dark_current_score is the share of items that fail this.
One JSONL line per call: hardware, prompt template id, seed, generation config, tokens, tps, peak memory, finish reason. The runs below are committed to the repo with their full records.
Before any model loads, two deterministic mocks bound the metric: a judge that always picks slot A scores 1.00, a judge that always answers Tie scores 0.00. Eight offline tests run in under a second, with zero downloads.
Three open-weight 4-bit judges, fully local (MacBook M3 Pro 18 GB, mlx-lm 0.31.3), greedy decoding with a fixed seed. The dataset is 12 hand-written pairwise items in three conditions: clear_gap (one answer plainly better), identical (byte-identical answers, so any pick is pure position signal), and paraphrase (same content, different wording).
| Judge (4-bit MLX) | clear_gap | identical | paraphrase | overall DC | gen tokens | wall |
|---|---|---|---|---|---|---|
| Meta-Llama-3-8B-Instruct | 0.25* | 0.25 | 0.50 | 0.333 | 4,459 | 170 s |
| Mistral-7B-Instruct-v0.3 | 0.00 | 0.00 | 1.00 | 0.333 | 2,725 | 106 s |
| Qwen2-7B-Instruct | 0.00 | 0.50 | 0.75 | 0.417 | 1,508 | 71 s |
Cells are per-condition dark current: the fraction of items whose verdict pair is not order-invariant. Lower is better. * One call answered correctly in prose but skipped the strict [[B]] marker, parsed as unknown rather than a flip.
All three judges read a genuine gap well: on parsed clear_gap verdicts they picked the objectively better answer in both orders 11 of 12 times.
Llama-3 drifts toward slot A even on byte-identical answers. Mistral is flawless until two answers merely say the same thing in different words, then it locks onto one slot on 4 of 4 paraphrase items. Qwen2 slot-locks on 2 of 4 identical pairs.
Llama-3 and Mistral share the same overall score, 0.333, and fail in different conditions. The per-condition breakdown is the finding, not the aggregate.
One fix mattered more than any prompt tweak: aligning the tokenizer's declared eos token with the mlx-lm stop set. Before it, every Llama-3 call burned the full 512-token budget: 12,288 generated tokens and 435 s versus 4,459 tokens and 170 s after, with identical verdicts.
The mock runner validates the whole pipeline with no model, no MLX and no network:
$ python3 eval_pipeline.py --runner mock --input data/sample_pairwise.jsonl # writes results/run.jsonl and results/summary.json $ PYTHONPATH=src python3 -m unittest discover -s tests # Ran 8 tests ... OK (~0.02 s)
A real judge run is a one-flag change:
$ uv venv && source .venv/bin/activate $ uv pip install -e ".[mlx]" $ python3 eval_pipeline.py \ --runner mlx \ --model mlx-community/Meta-Llama-3-8B-Instruct-4bit \ --input data/minislice_v0.jsonl \ --output results/llama3_8b_4bit_minislice_v0.jsonl
Per-condition breakdown from any run file: python3 scripts/category_breakdown.py results/*.jsonl
This is an independent pilot inspired by "LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation" (Usami et al.). It is not affiliated with or endorsed by the authors, and it does not reproduce the paper's full results.
Twelve hand-written items are a pilot, not a benchmark. They are enough to validate the harness and to show that per-condition breakdowns matter. The next milestone is a paper-aligned slice in the MT-Bench and AlpacaEval style. MIT licensed.