RCA

What I optimize for

Blame-free learning that drives measurable reliability gains.
Fast mitigation with clear communication.
Corrective actions that are verified, not just written.

My default approach

Treat RCA as a backlog generator, not a document archive.
Separate detection from prevention, and prevention from hardening.
Measure reliability in user impact, not internal comfort.

RCA philosophy

Every incident has a system behind it. I focus on mechanisms, guardrails, and incentives rather than individual mistakes.

Tools and artifacts I produce

Incident timelines and comms templates.
Corrective action trackers with owners and verification plans.
SLOs with clear error budgets and escalation triggers.
Runbooks for mitigation and fallback.

Incident workflow

Detect -> mitigate -> communicate -> learn -> prevent.
Capture the first hour with time-stamped notes and customer impact.
Convert learnings into backlog items with owners and verification.

Examples

txt

Incident timeline
- Detect: what failed and how we learned
- Mitigate: actions taken and time to stabilize
- Communicate: who was notified and when
- Recover: time to full service
- Follow-up: RCA owner, date, and next steps

txt

Five whys (done right)
1. Why did the user experience fail?
2. What system behavior made that possible?
3. What guardrail should have caught it?
4. Why was that guardrail missing or bypassed?
5. What change makes this class of failure unlikely?

txt

Corrective action format
- Issue: describe the failure mode
- Fix: change to prevent recurrence
- Owner: single accountable person
- Due: target date
- Verify: what signal proves it worked

Tradeoffs

Speed vs correctness: stabilize first, then rebuild confidence.
Guardrails vs innovation: build rails that unblock, not slow down.
Transparency vs noise: communicate impact, not every log line.

Signals I look for

Repeats drop over time for the same class of issues.
Mean time to mitigate improves with each incident.
Teams reference RCAs during planning and design reviews.

Anti-patterns

Vague root causes with no system changes.
Fixes without verification plans.
RCAs that are written but not socialized.