Method

vlm_macro_client.py attaches to the Qt screen and asks once each time a new piece appears.

  1. It draws the board as an image with 20 pixels per cell. Each piece type has its own colour.
  2. Along with the piece name (I, O, T…), it asks for "a spot to place it flat, without holes, so that lines are cleared".
  3. The answer is JSON with target_rot (0–3) and target_x (0–9).
  4. It sets the rotation, moves the piece to x and hard-drops it.

Results

We ran 2 games on the same engine.

  • 12 and 11 pieces, 0 lines, 50 and 46 holes
  • Asked 23 times, it kept the JSON format all 23 times.
  • target_x was 4 all 23 times. Only the rotation changed, between 1 (17 times) and 0 (6 times).
  • Each question took 5.7 seconds on average.

The result is the same as the macro DQN. It builds a tower in the middle and ends at around 12 pieces.

Why only x=4?

  • The picture has no column numbers or grid lines. It has to count 20-pixel cells to find a column, which is hard for a 3B model. In Gomoku too, when the VLM picked coordinates from the picture alone, it often picked occupied spots.
  • At the moment of asking, the new piece is in the hidden rows and does not appear in the picture. The piece is given only by name.
  • The prompt describes target_x as a "column", but the engine's x is the left edge of a 4×4 box. Even if it picks the right column, vertical pieces land one or two cells to the right. The leftmost-column problem from post 3 is still there too.

Instead of having the VLM make up coordinates, we switched to having code compute every possible spot and leaving only the choice to the model.

The actual code for this step

This is the question sent to the VLM.

vlm_macro_client.pypython · 12 lines
def get_macro_action(self, obs: np.ndarray, current_piece_name: str) -> tuple[int, int]:
    img = _grid_to_image(obs)
    prompt = (
        "You are an expert Tetris AI.\n"
        f"A new piece ({current_piece_name}) has just appeared at the top of the board.\n"
        "Look at the current board state and decide the best final resting position for this piece.\n"
        "Return your decision strictly as JSON with two keys:\n"
        '- "target_rot": an integer from 0 to 3 representing the number of clockwise rotations.\n'
        '- "target_x": an integer representing the target column (0-9 for a standard board, from left to right).\n'
        "Try to keep the stack flat, avoid creating holes, and clear lines if possible.\n"
        "Return ONLY the JSON string. Do not output anything else."
    )

It asks once when a new piece appears, and after that sends low-level actions until the target is reached.

vlm_macro_client.pypython · 11 lines
if piece_index > current_piece_index:
    # New piece spawned. Ask VLM for macro action
    piece_name = name_piece(piece_type)
    print(f"\n--- New Piece Spawned: {piece_name} (Index: {piece_index}) ---")
    current_piece_index = piece_index
    target_rot, target_x = agent.get_macro_action(grid, piece_name)
    target_x = max(0, min(target_x, w - 1))
    print(f"VLM Decision -> Target Rot: {target_rot}, Target X: {target_x}")

# Translate target macro state to next micro action
micro = next_micro_for_target(piece_type, cur_rot, cur_x, target_rot, target_x)

The next post is "5. Code computes the spots, the LLM only chooses".