The first code-level clue sits in `finishHand`. Chip delta is divided by 20 and clipped to the interval from -1 to +1.

src/model.cpp
const float reward = std::clamp(
static_cast<float>(chipDelta) / 20.f,
-1.f,
1.f
);A gain of +20 therefore produces +1, and a gain of +299 also produces +1. Losses beyond -20 are flattened in the same way. The learner loses ordering information between a modest win and a very large one, making it easier to optimize for reaching any positive endpoint than for efficient stack growth.

The next clue is credit assignment. The same terminal value is applied to every action stored in the episode.
src/model.cpp
constexpr float kRate = 0.05f;
for (const Step& step : episode_) {
for (int i = 0; i < kFeatures; ++i) {
float& weight = weights_[static_cast<size_t>(step.action * kFeatures + i)];
weight = std::clamp(weight + kRate * reward * step.features[i], -2.f, 2.f);
}
}
episode_.clear();
save();This does not distinguish how a fold, check, call, or raise contributed to the result. When an all-in ends a hand quickly, the trajectory is short and the same +1 can repeatedly reinforce only a few decisions. That does not prove a single cause, but it creates a clear structural advantage for an easily repeated short path.
The snapshots agree with this mechanism. For cheolsu, the all-in total is +0.818, versus -1.854 for check and -5.013 for call; the all-in stack feature is +0.226. Younghee's all-in value is near zero, but check and call are about -3.657 and -3.722, so all-in remains relatively attractive. Minsu instead leans toward `bbing`, not all-in.

Calling all three “the same all-in model” would therefore be inaccurate. The common issue is collapse toward a few actions, while the selected action and weight profile differ. A single coefficient is not the whole policy either: state features, action logits, and exploration jointly determine the move.
This diagnosis is deliberately limited. Clipping, uniform terminal credit, and short trajectories form a plausible mechanism consistent with the observed weights. Establishing causality requires controlled runs that vary clipping and credit assignment, then compare action distributions and long-run returns.
COMMENTS · CHAPTER 02
The Short Reward Shortcut Behind Policy Collapse 0 comments
No comments yet. Leave a question or anything you would like to add.
To post a comment, continue with Google and register a nickname and email address.
Continue with Google