Twelve days, read ring by ring
Career Compass is a privacy-first job-search co-pilot for Claude. It never watches its users, so I could only measure from what I had. This is how I found where it let people down, built a machine to measure it, climbed every axis on that machine, and closed out on 2.13.1. Each ring at right is one day; its width is the number of threads opened that day. Releases are marked in amber.
*Not like-for-like: the judge changed on Oct 4 and the test suite grew. See Act III.
before how
The product, the problem, the plan
Privacy first, by design
A job-search co-pilot inside Claude: fit verdicts, tailored résumés, cover letters, interview prep, offer reviews and a tracker. Your career history and applications live in local files on your own disk. There is no telemetry, not even opt-in usage counts. The cost of that choice: I can only measure from what I have, which is the directory dashboard, an opt-in feedback link, and an offline test suite.
Installs weren't turning into help
People installed it and didn't come back. In a job search one made-up metric on a résumé can cost someone the job, so "mostly true" isn't good enough. The goal: useful on the first try, truthful every time, and worth coming back to.
Listen, measure, climb, verify
Find the pain without watching anyone. Turn it into a judged eval suite. Climb one axis at a time and ship small releases. Finally, re-test everything as a stranger who has never seen the product.
Oct 1 – 3
202 installs, 18 people, and a greeting that broke on arrival
I opened the project by pasting the directory dashboard link. The plugin was healthy (0% load errors, 0 uninstalls). People installed it and then didn't use it. The first thread went looking for why.
Source: directory dashboard, data through Oct 1. Bars share one scale.
| Pain found on Oct 3 | What the user hit | Fix in 2.9.4 |
|---|---|---|
| Most installs came from Claude.ai, where the MCP server can't run | Skill said "say so plainly" and stopped. Dead end on try one. | No-tools mode: same work from pasted text |
| Skill only fired when you said "Career Compass" | Pasting a job posting did nothing | Triggers on any job-search ask |
| Setup before value | Build your whole career history first, help later | Answer first, offer to save second |
| No slash commands | Nothing obvious to type after install | /start, /fit-check, /interview-prep, /today |
The plugin worked. It just didn't know how to say hello.
12 instruments
Twelve ways to listen without watching
No telemetry is a promise in the README, so pain had to come from twelve other instruments: dashboards I pasted in, a judged eval suite, simulated reviewers, crash logs, competitors, research, and me using my own product the way a stranger would.
Oct 3 – 13
26 threads across seven lanes
Each bar is a thread, from its first message to its last. Amber ticks are releases. Tap or focus a bar to read what it did.
The climbs, release by release
Oct 3 – 14
The measuring machine, and the times it lied
Privacy first means the live numbers are coarse, and early on they were small (about 900 sessions a week). So decisions ran on an offline eval suite, approved on Oct 3 with "perfect, lets begin". Live dashboards checked whether the offline wins showed up for real people.
Judge calibration (Oct 4, PR #75)
The baseline said honesty was 57%. First guess: the judge was wrong. A careful re-audit said about 80% of those fails were real (drafts embroidering facts). Then a 40-reply hand audit graded the graders.
Agreement with hand audit. Memory cases went from 2/10 to 10/10; Opus caught 12 of 15 real fabrications.
Week one, live (data to Oct 6)
| Installs, all time | 780 |
| Installs per day after launch spike | ~125 |
| Daily active accounts | flat ~40 |
| Day-one retention (90-day CSV) | 9.4% (12/127) |
| Activation: Cowork / Claude.ai / Code | 44% / 15% / 15% |
| Uninstalls · disables of enables | 0 · 39 of 947 |
Verdict: installs were fine. Repeat use was the gap. Nobody was leaving; they were forgetting. Chat-suggestion installs doubled the day after 2.9.4 (25 → 48 → 52/day), the first visible discovery effect.
Ways the harness misled us, and the fix for each
6 checkpoints
The scoreboard, replayed
Each checkpoint below is a scored run. The dark tick on each bar is the 2.9.4 baseline. Axes not re-run at a checkpoint show as a dash.
before → after
Same question, different answer
Scores are a summary. These are the replies underneath them: what a job seeker actually got before a climb, and what they get now.
| Someone… | Before | After | Since |
|---|---|---|---|
| pastes a posting on Claude.ai | "The tools can't run here." End of conversation. | A fit verdict on the spot, from the pasted text alone | 2.9.4 |
| asks "what does this do?" | A menu of tools and a setup checklist | A few warm lines and one first step: paste a résumé and one posting | 2.9.9 |
| asks for a cover letter | "Adopted" quietly became "shipped". The letter invented why you want the job. | Your verbs and scope, kept. No motive you didn't give. Gaps marked [confirm: …], two at most. | 2.9.5 · 2.11 |
| tailors a résumé to a fintech posting | Borrowed the posting's skills ("PCI exposure") as if they were yours | Only skills you've said you have. The rest goes in the gap list. | 2.11 |
| researches a company | "Use web search" every time; remote work assumed | Company facts only from a source; unknowns stay unknown | 2.9.7 |
| gets an offer | A counter script that claimed market research it never did. Later: "companies expect it" stated as fact. | Counter by default, with one dated study and its real number. Nothing presented as a norm. | 2.13.0 |
| accepts an offer | No way to record it. The /today offer branch was dead code. | Offer fields, round outcomes, then thank-yous and withdrawals offered | 2.9.8 |
| comes back a week later | A blank slate | A pasteable card with a dated come-back line, plus a calendar reminder if they want one | 2.11 · 2.12 |
| runs /today in Desktop chat | "Tools aren't loaded, paste your card" | Reads the saved tracker read-only, says when it was last updated, ranks what's next | 2.13.1 |
Hiring research, built into the product
Sixty sources, each tagged primary, secondary or folklore. Only primary numbers made it into one vetted stat sheet the skills are allowed to quote.
| Finding | Source type | What Career Compass does with it |
|---|---|---|
| The only documented automatic rejection is employer knockout questions. AI graders sort; they don't reject. | Vendor docs | Fit checks look at knockouts first, then must-haves as meets / unclear / doesn't |
| Referrals reach interview at ~40%, cold applications at ~3% | Primary | A strong fit closes on a referral ask |
| Countering an offer raised pay ~12% in a randomized trial | RCT, 2025 | Counter is the default, with the study named |
| A written weekly plan raised offers ~30% | RCT | /week builds the plan |
| Fresh postings and tailored résumés each lift response rates | Primary | Find-roles leads with fresh postings, each with its own link |
| "75% of résumés rejected by ATS", "recruiters read for 6 seconds" | Folklore | Never quoted |
The advisory council's first flags
- No tool could record an offer, so the daily view's offer branch never ran.
- Saving a career section could silently drop entries; one broken file loaded as empty.
- The truth rule was missing from four tools that the changelog said it covered.
- Two first-reply rules contradicted each other, so a pasted posting got two questions back.
Five product-manager personas and one Anthropic-style technical reviewer, scoring on a fixed rubric. Retention and voice scored 2 out of 5. Every item that could be built shipped in 2.9.8 the next day.
2.13.1 · Oct 14
2.13.1, judged as a stranger
I asked for one big eval of everything, run as someone who has never met the product: fresh Claude profile, no memory, no name or email, an empty Career Compass folder. Sonnet 5.5 agent, Opus 5.5 judge, 3 runs per case, no ablation. Shipped as mcp #109 and plugin bundle #22.
| Axis | Passed | Score (95% CI) | vs 2.13.0 |
|---|---|---|---|
| Honesty | 184/192 | 96% (92–98) | +3 |
| Trust (no unasked writes) | 531/531 | 100% | = |
| Task quality | 131/150 | 87% (81–92) | +4 |
| Memory | 35/39 | 90% (76–96) | +8 |
| Retention behaviors | 33/36 | 92% (78–97) | +11 |
| First reply | 140/156 | 90% (84–94) | −2 |
| Activation (fires when it should) | 389/401 | 97% (95–98) | −1 |
| False fires | 0/90 | 0% (0–4) | = |
| Surface fit | 185/186 | 99% (97–100) | = |
| Voice | 84/84 | 100% | +1 |
Source: evals/release-2.13.1/summary.md. Memory and retention have wide intervals (39 and 36 checks), so read +8 and +11 as "probably up", not as precise gains.
Found and fixed during the final run
- Chat with a file tool said "I can't see your pipeline": the model globbed (Glob skips hidden folders) or guessed the home folder. Skills now pass the exact path with ~ as written.
- chat-where-was-i 0/3 → 3/3: one clause at most about a refused read.
- today digest for a laid-off engineer 1/3 → 3/3: ends with one offer, not three.
Measured against plain Claude
Career Desk, the direct rival, published a with/without honesty table. We measured ours on 2.11.2 (13 chat cases × 3, Opus judge) and published it as "Measured, not claimed":
| With plugin | Plain Claude | |
|---|---|---|
| Nothing untrue | 33/39 | 23/39 |
| Did the job | 33/39 | 16/39 |
Oct 14
Closed out, with the bark still growing
The project was closed after 2.13.1 shipped. Nothing is in flight. The offline axes are healthy. The live question from week one is still open. With no telemetry, the next answer comes from the next dashboard export.
| Then (Oct 1) | Week one (Oct 6) | Now (Oct 14) |
|---|---|---|
| 202 installs, 18 active | 780 installs, 157 active, DAU ~40 | ~5,000 installed (Oct 12 dashboard read, not yet checked against an export) |
| Uses in ~4% of sessions | Day-one retention 9.4% | Unmeasured. The day-two CSV pull never happened. |
| Honesty 57% (old judge) | Honesty 92% on 2.9.8 (Opus judge) | Honesty 96%, trust 100%, false fires 0/90 |
| No outside security grade | Unlisted on third-party registries | mcprush: Grade A security scan, trust score 97/100, zero threat findings; ~4,600 downloads all time |
| npm: ~15 downloads a day between releases | Not a plugin signal; the plugin bundles its own server | ~1,458 in the week to Oct 14, much of it release-day mirror traffic from 13 releases |
Still open on my side
- Public install link (claude.com/plugins/career-compass returns 404)
- Pull the 90-day CSV again to see if day-one retention moved after 2.12's come-back line and calendar reminder
- Weekly search-rank check (first audit row is still blank)
- Sign-ins: Smithery, mcp.so, MCPRush claim
- Repo descriptions, topics, Discussions; Cowork test on a clean machine
Known product gaps
- first-track-offers-briefing: skill doesn't fire in Code (0/3, carried by the tool instead)
- The cover letter tool can't take a pasted résumé
- Employer AI-use rules (write first, Claude refines) are only partly built in
- Real user reports should become eval cases; none have arrived yet
- Claude.ai chat is skills-only, so tools never attach there (15% activation)
The next click down
Close the live loop. One CSV pull answers whether 2.12–2.13 moved day-one retention off 9.4%. Every offline gain since Oct 9 is unconfirmed until then.
Climb the climbing process. Make each eval run lighter and faster, so a climb fits in an evening and runs every week.
Claude.ai is the biggest pool with the worst activation. The card, come-back line and Project tip were the bet. The CSV says if it paid.