Training settings

  • Algorithm: stable-baselines3 DQN. Learning rate 1e-4, replay buffer 100,000, batch 32, training every 4 steps, target network updated every 10,000 steps.
  • Exploration: the random rate drops from 1 to 0.02 over the first 20%.
  • Policy network: since observation values are 0–7, SB3 does not treat them as an image and picks MlpPolicy. The 220 board cells are flattened into one row.

Reward

Computed every time a piece locks.

  • Clearing lines: 1 line 1 point, 2 lines 3, 3 lines 10, 4 lines 30
  • A little more for placing lower (landing height×0.1)
  • -5 for each newly created hole
  • -2 for each row the stack height rises
  • Subtract bumpiness (sum of height differences between neighbouring columns)×0.02 and stack height×0.03

Results

Two low-level models remain: the 300,000-step model (live_latest.zip.bak) and the 370,000-step model trained further from it (live_latest.zip). The continued training aimed for 620,000 steps but stopped at 370,000.

We ran 20 games each with seeds 0–19 on the same engine. The model always picked the action with the highest value.

  • Random 7 actions: 17 pieces on average, 0 lines
  • DQN 300,000 steps: 22 pieces on average, 2 lines total over 20 games
  • DQN 370,000 steps: 20 pieces on average, 0 lines
Final boards of the 370,000-step DQN and of random play
Final boards of the 370,000-step DQN and of random play

Why it did not learn

  • It almost never sees a line reward. Pressing at random clears not a single line in 20 games. Clearing a line requires fitting several pieces together, which rarely happens during exploration. The signals coming into training are almost entirely hole and height penalties.
  • One piece takes more than 20 actions. Whether the spot was good only shows when the piece locks, so it is hard to tell which of the earlier moves and rotations did well.
  • A small MLP looks at 220 flattened numbers. It is hard for it to read piece shapes and positional relations, and it does not know the next piece either.

So we reduced actions to one per piece and had a VLM help with the reward.

The actual code for this step

This is the reward the engine computes when a piece locks.

tetris/src/tetris_engine.cppcpp · 41 lines
// Line clear reward (non-linear scaling)
double line_r = 0.0;
switch (cleared) {
    case 1: line_r = cfg_.line_reward_1; break;
    case 2: line_r = cfg_.line_reward_2; break;
    case 3: line_r = cfg_.line_reward_3; break;
    case 4: line_r = cfg_.line_reward_4; break;
    default: break;
}

// Height reward: lower placement = higher reward
double height_r = cfg_.height_reward_scale * landing_y;

const int holes_after = count_holes();
const int new_holes = holes_after - holes_before;
double hole_penalty = 0.0;
if (new_holes > 0) {
    hole_penalty = cfg_.hole_penalty_scale * static_cast<double>(new_holes);
}

// Stack height penalty: penalize increases of the board height (after line clears)
double height_penalty = 0.0;
if (cfg_.stack_height_penalty_scale > 0.0) {
    const int height_after = stack_height();
    const int dh = height_after - height_before;
    if (dh > 0) {
        height_penalty = cfg_.stack_height_penalty_scale * static_cast<double>(dh);
    }
}

// Surface shaping penalties (post line-clear)
double bump_penalty = 0.0;
if (cfg_.bumpiness_penalty_scale > 0.0) {
    bump_penalty = cfg_.bumpiness_penalty_scale * static_cast<double>(bumpiness());
}
double max_height_penalty = 0.0;
if (cfg_.max_height_penalty_scale > 0.0) {
    max_height_penalty = cfg_.max_height_penalty_scale * static_cast<double>(stack_height());
}

last_reward_ += line_r + height_r - hole_penalty - height_penalty - bump_penalty - max_height_penalty;

These are the reward magnitudes and the DQN settings.

config/rl_config.pypython · 18 lines
    # ── 보상/패널티 스케일 (C++ 엔진으로 전달) ───────────────────────────────
    # 한쪽으로 몰빵되는 탑쌓기를 억제하려면 아래 2개를 켜는 게 효과적
    stack_height_penalty_scale: float = 2.0   # 스택 높이 "증가분" 패널티 (권장 0.5~5.0)
    # bumpiness는 값 범위가 커서(최대 수백) 스케일을 너무 크게 주면 보상이 붕괴합니다.
    bumpiness_penalty_scale: float = 0.02     # 인접 열 높이 차이 합 패널티 (권장 0.005~0.05)
    max_height_penalty_scale: float = 0.03    # 절대 스택 높이 패널티 (권장 0.0~0.1)


@dataclass(frozen=True)
class DQNConfig:
    timesteps: int = 300_000
    learning_rate: float = 1e-4
    buffer_size: int = 100_000
    batch_size: int = 32
    train_freq: int = 4
    target_update_interval: int = 10_000
    exploration_fraction: float = 0.2
    exploration_final_eps: float = 0.02

The next post is "3. Macro actions chosen once per piece, and VLM rewards".