perftcheck

Install

Self-contained single-file binary, ~84 MB (the corpus is embedded). Published to GitHub Releases on every tagged build. No .NET runtime required on the target machine. The downloads below point at perftcheck-0.3.0.

$ curl -L -o perftcheck \
    https://github.com/Timmoth/grandchesstree/releases/download/perftcheck-0.3.0/perftcheck-<rid>
$ chmod +x perftcheck
$ ./perftcheck --engine ./your-engine

PS> Invoke-WebRequest `
    https://github.com/Timmoth/grandchesstree/releases/download/perftcheck-0.3.0/perftcheck-win-x64.exe `
    -OutFile perftcheck.exe
PS> .\perftcheck.exe --engine .\your-engine.exe

Usage

One required flag — --engine, pointing at a UCI executable. The default run walks every bundled position to depth 4 and writes perft-report.json. Source & full docs: Perft-Checker/.

$ ./perftcheck --engine ./stockfish

Exit codes: 0 all passed, 1 at least one failure, 2 bad args.

The corpus

142,953 positions scraped from forum posts, GitHub issues, EPDs, and chessprogramming.org — each carrying its source URLs, deterministic feature tags (en_passant_capture_possible, castling_white_kingside, etc.), full d1–d7 perft node counts, and the per-move root divides used by drill-down.

The pipeline that builds the corpus (search → fetch → extract → describe → perft → pack) lives in Perft-Checker/fen_corpus. d1–d6 totals are computed by GrandChessTree.Engine divide; d7 stragglers come from MPerft with nullmove counting on a small set of pathological multi-queen / mass-promotion positions.

Failure aggregation

After every run, the summary cross-tabulates the deterministic feature tags across the failure set. Tags present in ≥50% of failures are highlighted as strong root-cause signals.

Tags across the 17 failure(s) (top 20 of 23, ≥50% highlighted):
╭────────────────────────────────┬──────────┬──────────╮
│ tag                            │ failures │ fraction │
├────────────────────────────────┼──────────┼──────────┤
│ en_passant_capture_possible    │       16 │     94 % │ ← strong signal
│ side_to_move_black             │       14 │     82 % │ ← strong signal
│ castling_white_kingside        │        9 │     53 % │ ← strong signal
│ material_equal                 │        6 │     35 % │
│ middlegame_phase               │        5 │     29 % │
…

Same data lands in failureTags.countByTag / fractionByTag in the JSON report for programmatic analysis.

Drill-down: find the exact buggy move

When the engine's perft total disagrees with truth at depth N, the bug is usually one move in one sub-tree. Manually narrowing that down is the classic "perft divide → pick wrong root → apply → divide again" loop, repeated until depth 1 where the divide is the legal move list and the diff is the bug. --drill-down automates that loop.

Point it at a trusted reference engine — the GrandChessTree.Engine in this repo, which accepts the extended divide:<d>:<mb>:<fen>:<moves> command (per-move divide and the resulting FEN after applying an optional move list, in one round-trip):

$ ./perftcheck \
    --engine ./your-buggy-engine \
    --drill-down --ref-engine ./GrandChessTree.Engine

For every mismatch the runner:

1. Compares the test engine's divide at the failing position to the corpus's divide_d<N>.
2. Picks the first move whose count differs (missing → extra → wrong-count; missing moves are usually the actionable signal).
3. Applies that move via the reference engine, descends to depth N−1, asks the reference for the new expected divide, and compares again.
4. Stops at depth 1, where the diff between the test engine's and the reference's move lists is the bug — which moves the engine illegally generates, and which it failed to generate, at a single specific position.

Each failure in the JSON report gains a drillDown block:

{
  "drillDown": {
    "bugDepth":      1,
    "moveSequence":  ["e2e4", "g8f6", "f1c4"],
    "leafFen":       "rnbqkb1r/pppp1ppp/5n2/4p3/2B1P3/8/PPPP1PPP/RNBQK1NR b KQkq - 2 2",
    "missingMoves":  ["d8e7"],
    "extraMoves":    [],
    "wrongCount":    {}
  }
}

A bug-depth histogram is printed at the end of the run. A tall d=1 column means a pure move-generator issue; deeper bottom-outs point at make-move / unmake bugs (where the engine's d=1 divide after a fresh position fen agrees with truth, but its internal state after applying the same moves doesn't).

Cost: each mismatch adds roughly N reference-engine round-trips. For ~400 failures at depth 4, drill-down typically adds 5–15 minutes.

Common recipes

$ perftcheck -e ./engine \
    --quiet --report perft.json

$ perftcheck -e ./engine --quality high

$ perftcheck -e ./engine \
    --tag en_passant_capture_possible

$ perftcheck -e ./engine --fail-fast

$ perftcheck -e ./potential \
    --perft-command perft

$ perftcheck -e ./stormphrax \
    --perft-command perft \
    --bare-number-total

$ perftcheck -e ./engine \
    --include-edge-cases

$ perftcheck -e ./buggy \
    --drill-down \
    --ref-engine ./GrandChessTree.Engine

JSON report

Every run writes a single self-describing JSON file (default perft-report.json). Structured for programmatic ingestion — CI systems, spreadsheets, comparison scripts.

{
  "tool":      "perftcheck",
  "version":   "0.3.0",
  "engine":    "/abs/path/to/engine",
  "engineId":  "Stockfish 18",
  "startedUtc": "2026-05-30T12:00:00Z",
  "durationSeconds": 21.94,
  "options": {
    "depthMin": 1, "depthCap": 4, "timeoutSeconds": 30,
    "staticAnalysisFile": "embedded:corpus.gctc.gz"
  },
  "totals":   { "cases": 570293, "passed": 569869, "failed": 424 },
  "failures": [
    {
      "kind":     "mismatch",
      "fen":      "…",
      "depth":    3,
      "expected": 12345,
      "actual":   12344,
      "diff":     -1,
      "source":   "<embedded:corpus.gctc.gz>:8721",
      "tags":     ["en_passant_capture_possible", "side_to_move_black", …],
      "sourceUrls": ["https://github.com/X/Y/issues/123", …],
      "drillDown": { … }    // when --drill-down is on
    }
  ],
  "failureTags": {
    "failuresWithTags": 17,
    "countByTag":    { "en_passant_capture_possible": 16, … },
    "fractionByTag": { "en_passant_capture_possible": 0.94, … }
  }
}

Failure kinds: "mismatch" (node-count differs — expected/actual/diff populated), "timeout" (engine didn't respond within --timeout seconds), "error" (engine pipe closed or output unparseable; raw output captured in engineOutput, truncated to ~2 KB).

Engine compatibility

• Stockfish — works out of the box. Uses Nodes searched: N for the total and e2e4: 600 for divide lines.
• Komodo, Ethereal, Berserk, Lc0 — UCI-standard, work out of the box.
• Stormphrax — needs both --perft-command perft and --bare-number-total. Total prints as a lone integer on its own line; the go perft verb isn't recognised. Has a small known-divergence pocket (EP-blocks-check, pathological piece counts) covered by --include-edge-cases.
• StockDory — works out of the box; total uses comma thousands separators (Nodes searched: 1,234,567) and parses correctly.
• Viridithas — works out of the box; total is embedded inside a UCI info line (info depth N nodes K time T nps M) and parses correctly. Strictly rejects fullmove=0 and halfmove≥100 — pass FENs through a normaliser before testing if those quirks are in scope.
• Pawnocchio — works for most positions; total has trailing timing info (Nodes searched: N in Tms (M nps)) and parses correctly. Hangs on a small set of pathological positions — raise --timeout if you see timeouts.
• Jet — works out of the box; total is the bare Nodes: N form.
• Potential — accepts only the bare perft N form (no go prefix). Pass --perft-command perft. Divide lines are e2e4 600 (space, no colon) and parse correctly.
• GrandChessTree.Engine (this repo) — has a minimal UCI shim so it works as a test engine, and serves as the drill-down oracle via its colon-delimited extension command divide:<depth>:<mb>:<fen>[:<moves>] (apply moves + divide + return resulting FEN in one round-trip).
• MoveGen (this repo, MoveGen/) — adds UCI mode to the demo Program.cs.

A divide line is matched by the regex ^\s*([a-h][1-8][a-h][1-8][rnbqRNBQ]?)\s*[:\s]\s*(\d+)$, so both e2e4: 600 and e2e4 600 work, lowercase or uppercase promotion characters welcomed.