Technical incident RCA — from symptom to lasting correction

Separate service restoration from explaining the cause#

During an incident, limiting impact takes priority: switch traffic, restart a process, disable a faulty feature or roll back. These actions need not be the final solution. Their purpose is stabilisation.

RCA starts once the service is secured and analysis can proceed without immediate restoration pressure. Clearly distinguish:

• mitigation — what restored operation,
• cause — the mechanism that led to failure,
• lasting correction — what removes or controls that mechanism,
• prevention — what reduces its likelihood or detection time.

A restart that helped is a fact. It is not yet an explanation.

Build a timeline from clocks you can trust#

Base the timeline on logs, metrics, traces, tickets and operator actions. Before joining events, check time zones and NTP synchronisation. An application log in UTC and a customer report in IST can look like two separate failures.

For every entry, record:

• time and time zone,
• evidence source,
• event,
• impact or state change,
• confidence level.

For example:

Chronology helps identify correlation, but correlation alone does not prove causation.

Label facts, hypotheses and assumptions#

Many interpretations appear during an incident. “The database was overloaded” can be a fact if backed by CPU metrics and wait events, or a hypothesis inferred from application timeouts.

Use three categories:

• confirmed fact — supported by specific evidence,
• hypothesis — a possible mechanism requiring a test,
• assumption — temporarily accepted information because data is missing.

Define a falsification test for each hypothesis. If you suspect a firewall blocked RTP, captures on both interfaces should show whether packets arrived and where they disappeared. A hypothesis that no test could disprove is too general.

Describe the failure mechanism as a chain#

“Incorrect configuration” is a weak root cause. A better account shows how a particular value caused the effect:

new routing rule
→ did not cover forwarded calls
→ PAI contained an extension number
→ carrier rejected INVITE with 403
→ forwarded calls did not reach their destination

Connect the change or condition to the observed impact. Every arrow needs evidence or a clearly identified inference.

Often there is no single cause. A technical defect may have existed for a long time while a traffic change finally triggered an incident. Separate:

• the trigger,
• the latent defect,
• contributing factors,
• missing guardrails.

“Why?” should lead to a control, not a person#

The “5 Whys” method is useful if it does not end at “the administrator made a mistake”. People act within systems of processes, tools and information. More useful questions include:

• Why could the configuration be deployed without testing this path?
• Why did monitoring not detect the difference before the customer?
• Why did the test environment not reflect production?
• Why did rollback require manual reconstruction?
• Why did documentation omit a dependency?

The goal is to improve the system, not find the last person who touched its configuration.

Assess impact quantitatively and qualitatively#

Impact should answer business questions:

• how many users or customers were affected,
• which functions were unavailable,
• when the impact started and ended,
• whether data was lost or delayed,
• whether there were security or compliance risks,
• how fallback channels performed.

If exact figures are unavailable, provide a range and the estimation method. Avoid false precision. “Around 18–25% of call attempts on gateway B” is more honest than an unexplained figure calculated from incomplete CDRs.

A correction needs a regression test#

“Update the configuration” is incomplete. An action should specify:

• the precise change scope,
• an owner,
• a deadline,
• a success criterion,
• a pre-deployment test,
• a post-deployment test,
• a rollback method.

For a SIP fault, the test may verify specific headers on both legs. An application test should reproduce input that previously triggered the defect. Infrastructure may need a policy check in CI.

Divide actions into:

• immediate,
• short-term,
• strategic.

Not every strategic action needs immediate implementation, but it should have a deliberate decision and an owner.

Monitoring must detect impact and mechanism#

A 90% CPU alert can be useful, but users experience a broken function. After an incident, assess two layers:

• symptom monitoring — availability, errors, latency and call success,
• cause monitoring — queues, pool saturation, RTP loss, authentication failures and replica state.

A symptom alert tells you a customer has a problem. A cause alert helps you respond earlier. Both are needed.

An alert needs a runbook: what to inspect, which charts to open, how to recognise a false alarm and when to escalate. Otherwise monitoring only reports disorder faster.

RCA document structure#

A practical template:

1. Summary for non-technical readers.
2. Scope and impact.
3. Timeline.
4. Detection and response.
5. Technical facts.
6. Causal mechanism.
7. Contributing factors.
8. Mitigation.
9. Lasting correction and verification evidence.
10. Preventive actions with owners.
11. Unknowns and additional data needed.
12. Attachments: logs, traces, charts and commits.

The document should be precise enough for another person to follow the evidence, without copying thousands of log lines. Attach evidence and point to the relevant sections.

Warning signs of a weak RCA#

• the cause is “human error” without system analysis,
• mitigation and correction are not distinguished,
• no evidence connects the cause to the symptom,
• actions are only “monitor more closely” or “be careful”,
• owners and deadlines are missing,
• the correction lacks a regression test,
• the timeline mixes time zones,
• uncertainty is hidden rather than labelled,
• the same actions remain open weeks later without a decision.

An RCA is valuable when it changes how the system operates. Its best quality measure is not document length: it is whether a similar mechanism is detected earlier, blocked by a test or limited by architecture.