Learning by playing

We dropped order memorisation and let the model play the board itself. When the model makes a move, the server checks it against the answer, and if it is wrong, an oracle (code that knows the answer) plays the correct move instead. With the records accumulated this way, LoRA training continued every 3 games.

We ran 34 cycles over three days. The share of the model's moves that were correct was 2.0–3.6% per cycle, and 1,695 out of 62,842 (2.7%) overall. It did not improve however much we trained.

Finding a certain cell in one shot

Next, we had the model pick one 100% certain cell from the whole board directly (lora_cell). We ran 7 cycles, adding back the problems it got wrong, but its accuracy on 50 test problems was 6–16%. When asked to play 15 moves on real boards, it scored 0–33%.

We also tried a curriculum that split digits and cells and taught the easy part first. Digit accuracy fell from 34% to 26%, and cell accuracy never moved from 0%.

Accuracy over 34 cycles of trial-and-error training and 7 cycles of certain-cell finding
Accuracy over 34 cycles of trial-and-error training and 7 cycles of certain-cell finding

What the oracle taught

Around this time, I wrote down separately in solve.md how people solve sudoku. In that document I described the reason for the failure like this.

The oracle knows values with a probability below 20% and trains the VLM on them, so the trained results show no differences, and so the VLM seems to have no regularity at all about exactly why this answer turns out this way.

The oracle always knows the answer. It does not care whether that cell can be inferred from the current board. With answers mixed in that have no basis on the board, there was no rule for the model to learn. It was the same problem as the answer sheet in post 1.

The actual code for this step

This is one cycle of the loop that plays 3 games and then trains. It plays the games, extracts training data from the accumulated records and continues LoRA training.

vlm_play_train_loop.pypython · 38 lines
results, play_accuracy = run_games(
    session,
    games=args.games_per_cycle,
    difficulty=args.difficulty,
    max_steps=args.max_steps,
    verbose=args.verbose,
    save_images=None,
    game_id_start=state.game_id_next,
    accuracy=play_accuracy,
    human=args.human,
)
state.game_id_next += args.games_per_cycle
print_summary(results, play_accuracy)
print(f"\n사이클 {cycle} 시도정답률(리젝 포함): {play_accuracy.fmt()}")

# ...

total_lines = count_replay_lines(replay_path)
cycle_jsonl = cycle_sft_dir / f"cycle_{cycle:04d}.jsonl"
kept, new_total, by_outcome = export_trainable(
    replay_path,
    cycle_jsonl,
    since_line=state.replay_line_offset,
)

# ...

t0 = time.monotonic()
exit_code = run_training(
    dataset_jsonl=cycle_jsonl,
    base_model=args.base_model,
    adapter_dir=adapter_in,
    output_dir=args.output_dir,
    device=args.device,
    epochs=args.train_epochs,
    batch_size=args.batch_size,
    grad_accum=args.grad_accum,
)

The next post is "3. Turning human solving into rules".