Troubleshoot
Use when user reports an error, bug, or something not working. Search-first troubleshooting with diagnostic phase. Triggers: debug, error, broken, not working, failing, crash, exception.
npx skills add github:pantheon-org/tekhne --skill software-engineering--troubleshootSearch-first diagnostic workflow. Human executes commands.
⚠️ AskUserQuestion Guard
CRITICAL: After EVERY AskUserQuestion call, check if answers are empty/blank. Known harness bug: outside Plan Mode, AskUserQuestion silently returns empty answers without showing UI.
If answers are empty: DO NOT proceed with assumptions. Instead:
- Output: “⚠️ Questions didn’t display (known harness bug outside Plan Mode).”
- Present the options as a numbered text list and ask user to reply with their choice number.
- WAIT for user reply before continuing.
Workflow
0.Load → 1.Search → 2.Qualify → 3.Diagnose → 4.Investigate → 5.Persist → 6.Learn
0. Load Learnings
If a project-level learnings file exists (e.g., learnings.yaml in the project root or a known location), read it and apply known patterns before searching.
1. Search (do first)
80% of bugs solved online.
- WebSearch:
[error] [stack] [framework]on SO, GitHub, Docs, Reddit
"Type 'string | undefined' is not assignable to type 'string'" typescript
- Solution found → skip to 5.Learn
2. Qualify (2-3 questions)
AskUserQuestion:
- Stack? (language, framework, runtime)
- Environment? (local, container, cloud)
- Changed recently? (deploy, config, dependency)
3. Diagnose
See references/protocols/diagnose.md for details.
- Mental models: Check learnings file → WebSearch pattern → reason with 5 Whys/Fishbone
- Isolation: Wolf Fence (binary search), swap one variable, minimal repro
git bisect start
git bisect bad HEAD
git bisect good <last-known-good-sha>
- Root cause drill: 5 Whys, Fishbone 6 M’s
Pattern matches → suggest fix, skip OODA
4. Investigate (OODA)
Only if diagnosis inconclusive.
- Observe: User runs command, pastes output
- Orient: Analyze, update hypothesis
- Decide: Next command or confirm cause
- Act: Suggest fix (user executes)
Exit when root cause confirmed and fix verified.
5. Persist Thinking Artifact ⚠️ MANDATORY
MUST execute after root cause confirmed and fix verified. DO NOT skip. DO NOT wait for user to ask.
If a persistent thinking store is configured (e.g., via an environment variable pointing to a notes directory), write the diagnostic session there. Skip silently if no store is configured.
Content (required sections):
- Symptoms (as reported)
- Hypotheses tested (ordered list with result: confirmed/eliminated)
- OODA loops (if Phase 4 was entered)
- Root cause (as confirmed)
- Resolution (fix applied)
- Cynefin transition (if problem re-classified during diagnosis)
This is the active thinking trail — distinct from any learnings file which captures distilled, reusable conclusions.
6. Learn
After resolution, AskUserQuestion: “Save this learning?”
- Global → append to a shared learnings store
- Project → append with
scope: project:<name> - Skip
Philosophy
- Reproduce before you theorize — a bug that cannot be reproduced is a hypothesis, not a confirmed problem.
- Source location is the exit condition — every investigation ends with a specific file and line, not a module or service name.
- Distinguish symptom from cause — the first observable failure is rarely the root cause; follow the chain.
- Domain determines method — Complicated problems have known solutions; Complex problems need probes; do not mix the strategies.
When to Use
- When a user reports an error, exception, or crash with no clear cause
- When a system exhibits unexpected behavior that cannot be explained by recent changes
- When a bug is intermittent or flaky and cannot be reproduced consistently
- When log output or stack traces point to a symptom but not a root cause
- When a fix has been applied but the problem recurs or manifests differently
When Not to Use
- When the bug is already reproduced and the fix is obvious — apply the fix directly
- When the request is a feature request framed as a bug report — redirect to planning
- When the issue is a known, documented limitation with no workaround — communicate clearly
- When the environment is completely unavailable (no logs, no reproduction path, no user access) — gather access first
- When a postmortem or incident review is needed instead of a live diagnostic — use a dedicated retrospective workflow
Anti-Patterns
NEVER jump to a fix without reproducing the issue
WHY: Reproduction confirms you are solving the actual problem, not a symptom; untested fixes introduce regressions.
BAD: “Roll back the SDK upgrade” on the strength of a support ticket. GOOD: Reproduce the failure first, then change one variable and observe.
NEVER ignore the source code location step
WHY: Vague “somewhere in the auth module” diagnoses cannot be verified or code-reviewed.
BAD: “The problem is in the user service.”
GOOD: “Root cause at src/api/auth.ts:142: token expiry not caught.”
NEVER treat correlation as causation
WHY: Fixing the correlated symptom leaves the root cause intact.
BAD: Blame the deploy because errors started the same afternoon.
GOOD: Isolate with a swapped variable or git bisect before naming a cause.
NEVER skip the search step
WHY: About 80% of bugs have documented answers; skipping search wastes diagnostic cycles.
BAD: Ask five qualifying questions before running any search. GOOD: WebSearch the error, stack and framework first, then qualify.
NEVER persist a thinking artifact before root cause is confirmed
WHY: Premature writes capture incomplete hypotheses as conclusions, and misleading notes repeat the misdiagnosis in future sessions.
BAD: Write the artifact mid-investigation because someone wants a write-up. GOOD: Write it once the root cause is confirmed and the fix is verified.
NEVER proceed on empty AskUserQuestion answers
WHY: An empty answer means the question was never shown, so any assumption replaces the user’s reply.
BAD: Treat blank answers as “no recent changes” and continue. GOOD: Present a numbered text list and wait for the user’s choice.
NEVER enter OODA before diagnosis is exhausted
WHY: Pattern matching, isolation and root cause drilling resolve many bugs cheaply; OODA costs the user a round trip per loop.
BAD: Ask the user to run a new command as the first step. GOOD: Run mental models, isolation and the root cause drill first, and hand to OODA only when they are inconclusive.
NEVER run the user’s commands for them
WHY: The workflow is human-executed; the user holds the access and must see what runs in their environment.
BAD: Claim to have restarted the service or run the migration. GOOD: Give the command and ask the user to run it and paste the output.
NEVER mix Complicated and Complex strategies
WHY: Complex problems cannot be diagnosed in advance, so forcing a root-cause hunt produces confident but wrong answers.
BAD: Declare a cause for a problem whose three competing theories have all failed before. GOOD: Reclassify as Complex, note the Cynefin transition and route to a probe.
Usage Examples
Investigating a silent API failure:
# Skill prompts: reproduce → isolate → check logs → locate source
# Output: "Root cause at src/api/auth.ts:142 — token expiry not caught"
Triaging a flaky test:
# Skill prompts: classify domain (Clear/Complicated/Complex)
# Routes to probe skill if Complex domain detected
Diagnosing a performance regression after a deploy:
# Skill prompts: qualify stack/environment/recent changes → Wolf Fence bisect
# Output: "Regression introduced in commit abc123 — N+1 query in UserService.list()"
✅ Completion Checklist
Before responding to user, verify:
- Artifact written (or skipped if no store configured)
References
| Topic | Reference | When to Use |
|---|---|---|
| Isolation and root cause | references/reference.md | Wolf Fence, Swap One Variable, Minimal Repro, 5 Whys, Fishbone, OODA loop |
| Diagnose protocol | references/protocols/diagnose.md | Mental model matching, isolation sequence, OODA handoff conditions |
| Multi-source search | references/protocols/search-multi-source.md | Query construction, source priority and the no-result fallback |
| Handoff to framing | references/troubleshoot-to-frame-problem-llm.md | Passing an unresolved problem to frame-problem |
Explicit NEVER/WHY/consequence triplets with side-by-side bad/good examples. Generic warnings score low — concrete failure modes with real impact (security, correctness, performance) score high.
Runtime validation via eval scenarios proving the skill actually changes agent behaviour. Requires evals/ directory, instructions.json, ≥3 scenarios, and ≥80% instruction coverage in summary.json.
Appropriate rigidity for the skill type: strict rules for critical/security skills, balanced steps for process skills, flexible options for tool skills. Mismatched calibration (too rigid or too loose) reduces score.
Expert-only knowledge the AI couldn't derive on its own — domain-specific patterns, production gotchas, decision frameworks, and non-obvious tool behaviour. High scores mean <5% redundancy with what the model already knows.
A clear philosophy plus numbered step-by-step workflows with explicit entry/exit points. Should answer: what is the core principle, how to execute it, and when (and when not) to apply it.
Rich activation keywords and trigger scenarios in the description so the skill fires when needed. Best descriptions read like an exhaustive list of the tasks and concepts that should invoke this skill.
Copy-paste-ready code, complete runnable examples, real file paths and commands. Pseudocode and abstract descriptions score low — concrete, executable content scores high.
SKILL.md as a <100-line navigation hub pointing to focused references/ files. Content should be lazy-loaded on demand, not front-loaded. Includes actionable 'When to Use' conditions per reference.
Proper frontmatter, single-task focus, activation keywords, and cross-harness portability. Skills must work across 40+ agentic harnesses without hardcoded paths or agent-specific references.
Expert-only knowledge the AI couldn't derive on its own — domain-specific patterns, production gotchas, decision frameworks, and non-obvious tool behaviour. High scores mean <5% redundancy with what the model already knows.
A clear philosophy plus numbered step-by-step workflows with explicit entry/exit points. Should answer: what is the core principle, how to execute it, and when (and when not) to apply it.
Explicit NEVER/WHY/consequence triplets with side-by-side bad/good examples. Generic warnings score low — concrete failure modes with real impact (security, correctness, performance) score high.
Proper frontmatter, single-task focus, activation keywords, and cross-harness portability. Skills must work across 40+ agentic harnesses without hardcoded paths or agent-specific references.
SKILL.md as a <100-line navigation hub pointing to focused references/ files. Content should be lazy-loaded on demand, not front-loaded. Includes actionable 'When to Use' conditions per reference.
Appropriate rigidity for the skill type: strict rules for critical/security skills, balanced steps for process skills, flexible options for tool skills. Mismatched calibration (too rigid or too loose) reduces score.
Rich activation keywords and trigger scenarios in the description so the skill fires when needed. Best descriptions read like an exhaustive list of the tasks and concepts that should invoke this skill.
Copy-paste-ready code, complete runnable examples, real file paths and commands. Pseudocode and abstract descriptions score low — concrete, executable content scores high.
Runtime validation via eval scenarios proving the skill actually changes agent behaviour. Requires evals/ directory, instructions.json, ≥3 scenarios, and ≥80% instruction coverage in summary.json.
No eval scenarios available.
Troubleshoot Reference
1. Isolation Techniques
Wolf Fence (Binary Search)
Divide and conquer. Cut search space in half each step.
| Step | Action |
|---|---|
| 1 | Comment out / disable half the code |
| 2 | Bug gone? → It’s in removed half. Bug persists? → Other half |
| 3 | Repeat on guilty half until single line/block found |
Git variant: git bisect start → git bisect bad → git bisect good <commit> → Git finds guilty commit in O(log n)
Swap One Variable
When multiple suspects, change only one at a time:
- Works in env A but not B? → Diff A and B
- Works with data X but not Y? → Diff X and Y
- Works before commit Z? → Diff before/after Z
Minimal Reproduction
Reduce to smallest case that still fails:
- Remove code/config until bug disappears
- Add back last removed piece
- That’s your culprit
2. Root Cause Drilling
5 Whys
Keep asking “why?” until you hit root cause (usually 3-5 levels):
Problem: API returns 500
→ Why? Unhandled exception
→ Why? Null pointer on user.email
→ Why? User record has no email
→ Why? Migration didn't backfill
→ Why? Script assumed all users have email ← ROOT CAUSE
Rule: Stop when answer is actionable fix.
Fishbone (6 M’s)
For complex bugs with unclear cause, brainstorm across categories:
| Category | Questions |
|---|---|
| Man | Who touched it last? Training gap? Handoff issue? |
| Machine | Hardware? Memory? CPU? Disk? Network? |
| Method | Process flaw? Missing step? Wrong order? |
| Material | Bad input data? Corrupt file? Wrong format? |
| Measurement | Missing logs? Wrong metric? Alert gap? |
| Milieu | Environment diff? Config drift? External dep? |
Pick 2-3 most likely categories, investigate those first.
3. Assumption Surfacing
Rubber Duck
Explain the problem out loud, line by line:
- “This function should do X…”
- “First it gets Y from Z…”
- “Then it— wait, that’s not what it does”
Why it works: Forces you to articulate assumptions. Bug often surfaces mid-explanation.
Verify → Shorten → Reveal
| Phase | Action |
|---|---|
| Verify | Does bug actually exist as reported? Reproduce first. |
| Shorten | Make feedback loop fast (seconds not minutes). Write a test. |
| Reveal | Add logging at boundaries. Test each assumption explicitly. |
4. Investigation Loop (OODA)
When isolation + drilling doesn’t resolve:
OBSERVE → ORIENT → DECIDE → ACT → repeat
| Phase | Action | Example |
|---|---|---|
| Observe | Gather data | “Run X and paste output” |
| Orient | Form hypothesis | “This suggests Y because…” |
| Decide | Pick next action | “Let’s verify by checking Z” |
| Act | User executes | “Run this command: …” |
Exit when: Root cause confirmed + fix verified
Stuck? Go back to Search (step 1 in SKILL.md) with new keywords from investigation.
Troubleshoot to Frame-Problem Handoff
Thinking Trail
- Considered: {diagnostic approaches tried during troubleshooting}
- Rejected: {root causes eliminated}
- Surprised by: {problem is deeper than a simple degradation}
- Models used: Search-first diagnostics, systematic elimination
- Constraints discovered: {problem exceeds troubleshooting scope — needs re-framing}
Decisions
- Troubleshoot outcome: Stabilized but root cause requires broader change
- Domain shift: Complicated → Confused (re-classify for broader approach)
- Rationale: {fix applied but underlying issue needs architectural/process change}
Actions Taken
- Symptom addressed: {what was stabilized}
- Root cause identified: {deeper issue requiring re-framing}
- Learnings saved: troubleshoot learnings database updated
Output
Stabilized: {what was fixed} Root cause: {deeper issue identified} Re-frame needed: {why broader approach required}
Domain Transition
From: Complicated (troubleshoot) → To: Confused (frame-problem) Constraint shift: governing → unknown. Troubleshooting stabilized the symptom but revealed the root cause is a different class of problem. Need fresh classification.
For /frame-problem
- Context: Troubleshooting stabilized {symptom} but root cause is {deeper issue}
- What’s fixed: {immediate stabilization — do not re-troubleshoot}
- What remains: {root cause requiring different approach}
- New signals: {what troubleshooting revealed about the true problem}
- Do NOT: Re-diagnose the symptom — it’s stabilized. Classify the root cause.
Accumulated Context
Token guidance: target 300 tokens inline. For depth, use references — point to saved thinking files rather than embedding full content. Soft cap: 600 tokens inline per handoff. If you need more, move detail to a knowledge file and reference it. Accumulated cap: 800 tokens across a chain — compress to 200 at cap (keep: decisions, constraints, rejected paths). References do NOT count toward the cap.