--- name: claude-check version: 0.3.1 description: Test whether a Claude API endpoint or relay (中转站) really serves full-strength Claude (Fable 5, Opus 5 / 5.5, Sonnet 5 / 5.5) or a degraded / substituted model (降智、套壳、截断). Runs 10 freshly generated, program-graded questions and writes an HTML report with a degradation probability. Use when the user asks to check, verify, benchmark or "测降智" a Claude key, relay, base URL or the model behind Claude Code. --- # claude-check · Claude 降智检测 10 道每次全新生成的题目 → 程序判分 → 贝叶斯"降智概率" → HTML 报告。 Zero dependencies; needs Node.js 18+. All commands below run from this skill's folder. ## What it detects | # | Question | Catches | |---|---|---| | 01 | Long-context multi-needle retrieval (≈60k tokens, needles at 2/34/66/95% depth, revocations, a late correction) | context truncation / compression, weak long-range attention | | 02 | Register-machine trace with a loop (~45 dependent steps) | reduced thinking budget, smaller substitute | | 03 | Fresh 5×5 logic grid, solver-verified unique and minimal | weaker deduction | | 04 | Acrostic poem with exact per-line word counts, no repeated word, 22 checkable rules | weaker instruction following / planning | | 05 | Predict exact output of generated JS (Map order, bit ops, default lexicographic `sort`) | shallow code reasoning | | 06 | Multi-key sort of invented words | character-level precision | | 07 | Lattice paths through a checkpoint on an obstacle grid | careless multi-step computation | | 08 | Altered classic puzzles (the famous answer is wrong) | pattern-matching instead of reading | | 09 | 900 rule-generated lines (~7k tokens) | max_tokens caps, "..." elision, drift | | 10 | Identity self-report (asymmetric weight) | substitution by another vendor | API mode adds transport fingerprints: input-token metering vs. prompt size (truncation / tokenizer family), `stop_reason` vs. requested `max_tokens` (output caps), response id/model fields, thinking presence. ## Workflow ### 1. Collect the endpoint — never the key in chat Ask the user for: **base URL**, **protocol** (`anthropic` = `/v1/messages`, default; `openai` = `/v1/chat/completions`) and the **claimed model id** (`claude-fable-5`, `claude-opus-5-5`, `claude-opus-5`, `claude-sonnet-5-5`, `claude-sonnet-5`; relay spellings such as `claude-opus-5.5` are fine). The key must come from the environment. Tell the user to set it in *their own* terminal; do not ask them to paste it into the conversation and never echo it: ```bash export CLAUDE_CHECK_API_KEY=... # or ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN / OPENAI_API_KEY ``` If the user wants to test the relay that Claude Code itself is using, the existing `ANTHROPIC_BASE_URL` + `ANTHROPIC_AUTH_TOKEN` / `ANTHROPIC_API_KEY` are picked up automatically. ### 2. Show the cost, get confirmation ```bash node scripts/check.mjs run --model claude-opus-5-5 --base-url https://relay.example.com --dry-run ``` Report the estimate to the user (≈69k input + ≈50k output tokens; list price ≈ $0.6 Sonnet, ≈ $1.3 Opus 5.5, ≈ $3 Fable 5). Real calls are billed by their endpoint, so **wait for an explicit yes** before step 3. ### 3. Run ```bash node scripts/check.mjs run --model claude-opus-5-5 --base-url https://relay.example.com --yes ``` Options: `--lang zh-CN` (Simplified Chinese report and messages; default English — pass it whenever the user writes in Chinese), `--protocol openai`, `--context small|standard|deep` (≈15k / 60k / 400k tokens for Q1), `--seed S` (reproduce a run), `--only 1,9` (subset), `--concurrency 3`, `--out DIR`. A run takes a few minutes. Progress goes to stderr; the Markdown summary goes to stdout. **Relays that name the model differently or mount the API elsewhere.** `--model` is what the results are judged as. If the relay calls it something else, add `--upstream-model NAME`: that name is what gets sent. `--path /content/models/{model}/generations` replaces `/v1/messages` or `/v1/chat/completions` (`{model}` becomes the upstream name), and `--header 'Name: value'` adds request headers (up to 8, repeatable). A header whose value is a secret goes in the environment as `CLAUDE_CHECK_HEADERS='{"x-api-key":"..."}'`, not on the command line. The checker keeps the headers it owns (`Authorization`, `Content-Type`, `Host`, and the key header it was told to use); header values are redacted from every output and the report shows header names only. ### 4. Present the result - Lead with the **降智概率** and verdict (满血 < 15% ≤ 基本正常 < 40% ≤ 疑似降智 < 70% ≤ 高度疑似降智), then the 诊断 bullets. - Give the path to `report.html` (offer to open it: `open` on macOS, `xdg-open` on Linux) and `report.json`. - If the verdict is 疑似/基本正常 with only one or two misses, suggest one more run with a new seed before concluding: single runs are noisy. - If the report says the baseline is uncalibrated, say the probability uses expert priors. ### Manual mode (chat apps, no API key) ```bash node scripts/check.mjs export --model claude-opus-5-5 --out ./cc-test # writes questions/q01.md … q10.md # user pastes each question into a NEW chat, pastes the full reply into answers/qNN.md node scripts/check.mjs grade --dir ./cc-test ``` Add `--lang zh-CN` to `export` for a Chinese report (grade reuses it). Manual mode defaults to `--context small` so Q1 fits in a chat box. No transport fingerprints are available. ## Rules - **Do not answer the questions yourself** or "help" the tested model. The point is to measure the endpoint. - Do not edit graders, expected answers or the baseline to change a verdict. - Never print, log or store the API key. Reports contain only the endpoint hostname. - One run proves one sample. Relays can route requests differently over time; say so when relevant. ## Files - `scripts/check.mjs` — CLI (`run`, `export`, `grade`, `baseline`). - `scripts/lib/questions/` — the 10 generators and graders. - `scripts/lib/scoring.mjs` — likelihood-ratio scoring and diagnosis. - `scripts/hosted.mjs` — hosted mode: SandBase Lab starts this inside a sandbox; it claims the run configuration from Lab with a one-off token and pushes progress and the report back. Not for local use. - `assets/baselines/claude.json` — official pass rates and tokenizer reference (maintained by SandBase Lab with `baseline`). - `references/methodology.md` — scoring math, calibration and limitations.