5. Fixing the evaluation set before training

What counts as "better"?

A falling training loss does not mean real games got better. A Go AI must read coordinates, play legal moves, not pass in the middle of a fight and not fill its own eyes needlessly. Before changing the model, measure these items on the same evaluation set.

Step 1: Separate the evaluation items

The coordinate test measures whether stone positions are read correctly. The rule test counts the share of illegal moves the model produces. The game test plays a fixed opponent with colours swapped and records captures, when passes happen and why games end. If different failures are merged into a single win rate, you cannot tell what to fix.

1. Coordinate accuracy in the four quadrants
2. Illegal-move rate on the first answer and on retries
3. Number of early passes before move 80
4. Directional bias that keeps choosing the same column or corner
5. Captures and final results of games with colours swapped

Step 2: Keep the baseline model's results

Do not run only the new model; store the current production model's results in the same JSON format. Write down the evaluation data version, model name, prompt and rule engine commit, so the numbers can be reproduced later. If there is randomness, fix the seed and generation settings too.

Step 3: Mark the limits of short games

A provisional count after stopping at move 80 is useful as a quick regression check, but it cannot be read as an official win rate or rank. Life-and-death and the endgame are still unresolved. Short evaluations look at illegal moves, early passes and obvious tactical losses; comparing playing strength requires collecting enough complete games and judging separately.

Step 4: Turn failures into the next data

Failures that repeat in evaluation are saved together with the board state, the model's raw response and the rejection reason. Coordinate errors go to the input tests, futile ladder escapes to tactical examples and early passes to examples of ongoing games. Do not throw every failure into training; separate the problems the rule code can reliably block from those the model has to learn.

A good evaluation sheet is not a score for praising the model but a map that shows where to fix next.

Actual records

While training the LoRA in five rounds, we ran the same coordinate test again after each round. In r1, reading a stone's position as a coordinate was 100%, but answering which stone sits at a given coordinate was 78.3%. Looking at a single item would have missed this gap.

The actual code for this step

This code aggregates the coordinate test results into four areas: top left (TL), top right (TR), bottom left (BL) and bottom right (BR). It keeps the per-area accuracy alongside the overall accuracy, so directional bias is visible immediately.

def _quadrant(r: int, c: int) -> str:
    if r < 9.5 and c < 9.5:
        return "TL"
    if r < 9.5 and c >= 9.5:
        return "TR"
    if r >= 9.5 and c < 9.5:
        return "BL"
    return "BR"

def summarize(results: List[TrialResult]) -> Dict[str, Any]:
    by_kind = Counter()
    ok_kind = Counter()
    by_q = Counter()
    ok_q = Counter()
    for r in results:
        by_kind[r.kind] += 1
        by_q[r.quadrant] += 1
        if r.ok:
            ok_kind[r.kind] += 1
            ok_q[r.quadrant] += 1
    total = len(results) or 1
    return {
        "n": len(results),
        "accuracy": round(100.0 * sum(1 for r in results if r.ok) / total, 1),
        "by_kind": {
            k: {
                "n": by_kind[k],
                "ok": ok_kind[k],
                "acc": round(100.0 * ok_kind[k] / max(1, by_kind[k]), 1),
            }
            for k in sorted(by_kind)
        },
        "by_quadrant": {
            q: {
                "n": by_q[q],
                "ok": ok_q[q],
                "acc": round(100.0 * ok_q[q] / max(1, by_q[q]), 1),
            }
            for q in ("TL", "TR", "BL", "BR")
        },
        "avg_ms": round(sum(r.ms for r in results) / total, 1),
    }

Model used: Qwen3-VL-8B-Instruct + LoRA adapter (lora_mix_qwen_r5). The full source and a description of the model are in the post "8. The full source code and the models used".