1. Wrapping a C++ Tetris engine as a reinforcement learning environment

Goal

Do reinforcement learning with our own C++ engine instead of external environments such as gym-tetris or a NES ROM. Training runs without a screen, and the Qt screen is attached only to view results.

The engine

• The board is 10 cells wide and 26 tall. The top 4 rows are hidden rows where pieces spawn, so only 22 rows appear on screen and in observations.
• Pieces are drawn with a 7-bag. All seven kinds are shuffled and dealt once each, so no piece is missing for long.
• On rotation, it tries 7 positions in order, from the original spot to spots pushed left, right and up, and places the piece in the first one that fits.
• After a move, rotation or no-op, the piece drops one row on its own. So the number of actions available per piece equals the board height.
• The reward is computed the moment a piece locks. The reward makeup is covered in the next post.

C API and Gym environment

The engine is a C++ class, so Python cannot call it directly. It is wrapped in 21 extern "C" functions and built as a shared library (libtetris_c_api.dylib), which Python loads with ctypes. This library builds without Qt.

LocalTetrisEnv builds a gymnasium environment from these functions.

• Observation: 22×10 cells. Empty cells are 0 and blocks are 1–7 by kind. Locked blocks and the moving piece are not distinguished.
• Actions: 7. No-op, left, right, rotate clockwise, rotate counter-clockwise, soft drop one row, hard drop.
• Reward: the value computed by the engine, used as is.

The engine knows the next 5 pieces, but they were not included in the observation.

Qt screen and TCP

Launching the Qt app (tetris_rl) with --rl-port 5555 opens a TCP server. The app sends the board state as one line of JSON, and the client returns an action number on one line. The JSON contains the board, the current piece's kind, position and rotation, and the list of next pieces. This channel is used to run a trained model on screen or to attach the LLM and VLM clients that appear later.

The actual code for this step

The C API holds an engine in a single handle, takes an action number and advances one step.

struct TetrisHandle {
    TetrisEngine engine;
    std::vector<uint8_t> render_grid;
    explicit TetrisHandle(const TetrisEngineConfig& cfg) : engine(cfg) {}
};

// Control
void tetris_set_seed(TetrisHandle* h, uint64_t seed) { h->engine.set_seed(seed); }

void tetris_reset(TetrisHandle* h) { h->engine.reset(); }

// Step returns 1 if alive, 0 if game over
int tetris_step(TetrisHandle* h, int action) {
    const auto a = static_cast<TetrisAction>(static_cast<int8_t>(action));
    const bool alive = h->engine.step(a);
    return alive ? 1 : 0;
}

The Python environment cuts the 4 hidden rows from the board filled by the engine and uses the rest as the observation.

def _obs(self) -> np.ndarray:
    written = int(self._lib._dll.tetris_fill_render_grid(self._h, self._buf, self._w * self._hgt))
    if written <= 0:
        raise RuntimeError("tetris_fill_render_grid() 실패")
    arr = np.frombuffer(self._buf, dtype=np.uint8, count=self._w * self._hgt)
    arr = arr.reshape(self._hgt, self._w)
    visible = arr[self._hidden :, :]  # drop hidden rows
    return visible[:, :, None]

def reset(self, *, seed: int | None = None, options: dict[str, Any] | None = None):
    super().reset(seed=seed)
    if seed is not None:
        self._lib._dll.tetris_set_seed(self._h, int(seed))
    self._lib._dll.tetris_reset(self._h)
    return self._obs(), {}

def step(self, action: int):
    alive = int(self._lib._dll.tetris_step(self._h, int(action)))
    reward = float(self._lib._dll.tetris_last_reward(self._h))
    terminated = (alive == 0) or (int(self._lib._dll.tetris_game_over(self._h)) == 1)
    truncated = False
    info = {
        "lines_cleared_total": int(self._lib._dll.tetris_lines_cleared_total(self._h)),
        "piece_index": int(self._lib._dll.tetris_piece_index(self._h)),
        "stack_height": int(self._lib._dll.tetris_stack_height(self._h)),
        "bumpiness": int(self._lib._dll.tetris_bumpiness(self._h)),
        "holes": int(self._lib._dll.tetris_holes(self._h)),
    }
    return self._obs(), reward, terminated, truncated, info

The next post is "2. DQN for 370,000 steps, zero lines cleared".