Macro actions

MacroPlanningTetrisEnv places one piece all the way down with a single action. The action number is target column×4 + target rotation, 40 in total. Inside the environment, the rotation is set first, then the piece is moved left or right and hard-dropped. More than 20 decisions per piece shrink to one, and the reward for clearing lines attaches directly to the action that chose that piece's spot.

Results

The macro DQN (dqn.zip) was trained for 300,000 steps, which is 300,000 pieces. These are the results over 20 games with the same seeds.

  • Random macro: 25 pieces on average, 0 lines
  • DQN macro: 12 pieces on average, 0 lines

It ends earlier than random. Over 20 games it chose only 5 kinds of actions, and action 13 (column 3, rotation 1) and action 23 (column 5, rotation 3) make up 78%. It drops pieces in two spots in the middle regardless of the piece type.

The action encoding blocked the leftmost column

Piece shapes are drawn inside a 4×4 box, and the engine's x is the left edge of this box. A vertical I sits in the third cell of the box, so placing it in the leftmost column requires x to be -2. Macro actions clamp the target column to 0–9, so such spots cannot be chosen.

There are 162 places in total where the seven pieces can go, and 151 of them can be chosen with the 40 actions. The 11 missing ones are placing a vertical I, an O, or a vertical T, J or L in the leftmost column (8) or the second column (3).

We put the same restriction on the heuristic planner that appears in post 5. Using every spot, it clears 370 lines on average over 20 games; using only spots with x of 0 or more, 7.9 lines. Once the leftmost column becomes a well, no piece can fill it, and that column stays empty to the end.

The MoE that ranks candidates by rules (moe_macro_policy.py) also ran on top of these 40 actions and managed only 12.4 lines on average. The MoE's final board also has the leftmost column mostly empty.

VLM rewards

We tried to fill the scarcity of line rewards with a VLM. LLMRewardWrapper draws the board as an image every 10 steps, shows it to Qwen2.5-VL-3B and receives a score from -1 to 1 and the next action as JSON. The training reward is the engine reward + 0.2×score. The other 9 steps reuse the previous score. The recommended action is only logged, not used. A Qwen2.5-3B (GGUF) judge that reads a text board is included with the same structure.

No results from training with this reward turned on remain. The macro DQN was saved before the VLM reward code existed, and there is no record of which reward settings the 370,000-step low-level model was trained with.

So we tested the judge on its own. After advancing a board partway with the heuristic, we showed it, on the same board, the result of placing the piece at the top-ranked spot and at the bottom-ranked spot. The bottom-ranked spot creates 7.0 new holes on average.

  • Average score over 15 pairs: top spot -0.77, bottom spot -0.79
  • Pairs where the top spot scored higher: 4; equal: 10; lower: 1
  • Every score received was either -0.8 or -0.7

Adding it at 0.2× just attaches the same penalty of about -0.15 to almost every step; it conveys nothing about whether a spot is good or bad.

The actual code for this step

This unpacks one macro action into low-level actions and places one piece. The line that clamps the target column to 0–9 is what blocked the leftmost column.

macro_planning_env.pypython · 40 lines
def step(self, action: int):
    target_rot = int(action) % 4
    target_col = int(action) // 4
    if target_col < 0 or target_col >= self._w:
        target_col = int(np.clip(target_col, 0, self._w - 1))

    start_pi, _, _, _, _ = self._piece_state()
    total_reward = 0.0
    terminated = truncated = False
    info: dict[str, Any] = {}

    for _ in range(self._max_micro_steps):
        pi, ptype, px, py, prot = self._piece_state()
        if self._micro._lib._dll.tetris_game_over(self._micro._h):
            obs = self._micro._obs()
            terminated = True
            info = {
                "lines_cleared_total": int(
                    self._micro._lib._dll.tetris_lines_cleared_total(self._micro._h)
                ),
                "piece_index": int(self._micro._lib._dll.tetris_piece_index(self._micro._h)),
                "macro_steps": _ + 1,
            }
            return obs, total_reward, terminated, truncated, info

        if pi > start_pi:
            break

        if self._controller is not None:
            # RL low-level controller (Discrete(7))
            micro_a, _ = self._controller.predict(obs, deterministic=True)
            micro = int(micro_a)
        else:
            micro = next_micro_for_target(ptype, prot, px, target_rot, target_col)
        obs, r, terminated, truncated, info = self._micro.step(micro)
        total_reward += float(r)
        if terminated or truncated:
            info = dict(info)
            info["macro_steps"] = _ + 1
            return obs, total_reward, terminated, truncated, info

This is the part that adds the VLM score to the reward.

llm_reward_wrapper.pypython · 15 lines
def step(self, action: int):
    self._last_action = int(action)
    obs, reward, terminated, truncated, info = self.env.step(action)

    # LLM judge uses obs after action
    next_action, score = self.teacher.judge_and_suggest(obs, self._last_action, dict(info))
    shaped = float(reward) + float(self.cfg.alpha) * float(score)

    info = dict(info)
    info["llm_score"] = float(score)
    info["llm_next_action"] = int(next_action)
    info["reward_env"] = float(reward)
    info["reward_shaped"] = float(shaped)

    return obs, shaped, terminated, truncated, info

The next post is "4. Letting the VLM pick the piece's spot directly".