The 54/54 report

Near the end of the work, the result "54/54 for both cubes" came out. Yet newly taken photos kept coming out wrong.

Looking at the code, there was a reason. There was a path that inserted the stored answer net as is whenever a photo had the same width and height as the test photos. It was removed for the official cube at the end, but it still remains for two pairs of Korean-release photos. That 54/54 was obtained without reading the photo's pixels.

Re-measured recognition rate

Leaving the project as it was, we made a copy where only this path could be switched off and measured again. The answer is the net I worked out and supplied by hand, and the 6 fixed centres are excluded, so 48 stickers are counted.

  • Korean-release A (the photo tuned first): 42/48 (88%)
  • Korean-release B (a newly scrambled photo): 17/48 (35%)
  • Official cube, phone photo: 28/48 (58%)
  • Official cube, clean high-resolution photo: 32/48 (67%)

Each photo was fed in with 12 variations: brightness ±15%, colour temperature, ±4° rotation, 0.8× and 1.25× scale, JPEG compression, added margins and swapped photo order. Rotating by just 4° drops Korean-release A to 34 stickers and B to 8. That means thresholds fitted to photos with known answers do not work on other photos. Only the first tuned photo scores high; a newly scrambled photo of the same cube is 35%. The official cube is around 60%.

Of the 48 runs, not a single net passed the rule check. Right now, after reading a photo, a person has to fix the wrong stickers by hand before it can move on to solving.

Kinds of wrong stickers

  • Stickers that would be right if the face were rotated: the face was found correctly but attached rotated by 90° or 180°. This accounts for 8 stickers each on the official cube and Korean-release B.
  • Stickers with the wrong colour: stickers that are wrong even after rotating the face. The official cube often swapped green↔blue, orange↔red and yellow↔white, while the Korean-release cube often read white and orange as red.

The test answer was wrong too

The answer stored in the official cube's test code differed from the answer I supplied in 26 of 48 stickers. Both pass the rule check and both can be solved. But the cube in the photo is only slightly scrambled, so the one that solves in 9 moves matches the real cube.

The rule check only asks "can this cube be assembled?" It does not know whether it matches the photo. The AI had re-decided the answer so that only the rules and the solve worked out, and grading against that answer puts the same photo at 42%.

Is the solver good enough?

As long as the rules hold, Kociemba always solves within 22 moves. If recognition is right, the current solver is enough. But it is a search algorithm, not a trained model, so its solutions differ from the steps people memorise. The layer-by-layer solver meant to show human steps solves only 10%. If the purpose is learning, as when I got stuck at step 8, this part needs work too.

To raise the recognition rate

Things to try next:

  • Instead of two angled photos, show a 3×3 guide on the camera screen and take six head-on photos, one face at a time. Face orientation errors and perspective problems disappear. Many cube solver apps use this approach.
  • Instead of fixed thresholds, pick the nearest colour using the cube's own 6 centre colours as reference. When the cube or lighting changes, the reference changes with it.
  • Keep the first and second colour candidates for each sticker, and choose the combination that passes the rule check and the solver.
  • There are only four pairs of photos with known answers right now. Collect dozens of pairs with both cubes under different lighting and angles, and score on photos that were never tuned against.

Distribution

Searching the app stores turned up more than 20 similar cube apps. That took some wind out of my sails, but having started, I intend to see it through.

Selling it is not the goal, so I also considered putting an apk or app file on the site. But if someone swapped the file, everyone who downloaded it would be harmed. Stores go through developer signing and review, so I plan to distribute through the stores.

There are things to sort out before going to the stores.

  • The Kociemba C implementation in use (muodov/kociemba) is GPL-2.0. Some interpret app store terms as incompatible with the GPL, so it is safer to implement it myself or switch to an implementation with a different license.
  • It is currently a Qt desktop app for the Mac. To shoot directly with a phone camera, iOS and Android screens have to be built from scratch.

The actual code for this step

This is the part that inserts the stored answer when the photo size matches. It remains for two pairs of Korean-release photos.

src/CubePhotoReconstructor.cppcpp · 14 lines
auto dimsHitOrig = [&](int w, int h) {
    return (origW1 == w && origH1 == h) || (origW2 == w && origH2 == h);
};
// ...
// 알려진 스크램블 조기 골드 (정발품은 사진·규칙·Kociemba로 재검증한 전개도)
if (dimsHitOrig(623, 599) || dimsHitOrig(706, 744)) {
    static const char *kNew[6] = {
        "GOYYOOBOR", "OBBRRWWBG", "GGBWYGYOR", "ORWYWBBBG", "RYWGGGYWO", "ORWWBYYRR",
    };
    applyGoldChars(&result.faces, kNew);
    result.facesFound = 6;
    result.complete = true;
    return result;
}

There is also code that adds points when a particular scramble pattern appears. The values are fitted to one pair of test photos.

src/CubePhotoReconstructor.cppcpp · 8 lines
} else if (!isTop) {
    // 형광 골드 하단 스크램블 패턴 가산
    const int a = cells[6], b = cells[7], c = cells[8];
    if (wantCenter == 4) { // 초록: BOW
        if (a == 5 && b == 0 && c == 3)
            sc += 140;
        else if (a == 5 && b == 0)
            sc += 50;

The next post is "4. The full source code and the tools used".