Purpose

This runbook guides on-call responders through diagnosing and resolving a failed health check on the Order Processing API, restoring service availability with minimal downtime.

When to Use This Runbook

• Automated monitoring flags the service as unhealthy or unreachable.
• Customer-facing errors are reported that trace back to this service.
• Health check endpoint returns a non-200 status code for more than 2 consecutive checks.

Severity Levels

Level	Criteria	Response Time
Sev 1	Full service outage, customer-facing impact	Immediate
Sev 2	Degraded performance, partial impact	15 minutes
Sev 3	Isolated failure, no customer impact yet	1 hour

SLA

Milestone	Target
Acknowledge alert	5 minutes (Sev 1/2), 15 minutes (Sev 3)
Root cause identified or escalated	30 minutes from acknowledgment
Service restored	1 hour from acknowledgment (Sev 1)

Prerequisites

• Access to the monitoring dashboard (New Relic or equivalent)
• Access to deployment history / CI-CD pipeline logs
• PagerDuty on-call access

Diagnostic Steps

1. Confirm the alert: check the monitoring dashboard for current service status and recent error rate trends.
2. Check for a networking outage: confirm whether internal networking, DNS, or load balancing is affected.
3. Check for a vendor outage: confirm status pages for any third-party vendors this service depends on.
4. Check recent deployments: review deployment history for the last 24 hours. A recent deploy is the most common root cause.
5. Check upstream dependencies: confirm whether any services this API depends on are also reporting issues.
6. Review application logs for error patterns (timeouts, null references, authentication failures).
7. Determine whether the issue is isolated to one instance/region or affects the service broadly.

Resolution Steps

1. If caused by a recent deployment: initiate rollback to the last known-good version.
2. If caused by an upstream dependency, networking issue, or vendor outage: escalate to the owning team or vendor and monitor for recovery.
3. If caused by resource exhaustion: scale the affected service and confirm recovery via the health check endpoint.
4. If applicable, restart the affected service(s) to clear the faulted state.
5. Verify the health check returns a 200 status and error rates return to baseline.

Rollback Procedure

Use deploy rollback --service order-processing-api --to-version [last-known-good] via the deployment pipeline. Confirm rollback completion in the CI/CD dashboard, then re-run the health check.

Escalation Path

1. On-call SRE (first responder)
2. Service owner / team lead
3. Incident commander (contacted for any incident, regardless of severity, and briefed on the identified cause once known, unless the issue is confirmed to be a simple bug)

Post-Incident

• Log the incident timeline and resolution in the incident tracker.
• Schedule a postmortem for Sev 1 and Sev 2 incidents.
• The on-call SRE submits a Root Cause Analysis (RCA) for approval by the team leader within 5 days of the incident. The RCA includes the incident timeline, resolution, and suggestions for change.
• A corrective actions document is created to document and track any necessary long-term changes identified by the RCA.
• Update this runbook if the resolution path uncovered a new failure mode.

Purpose

This runbook guides the deployment owner and on-call engineer through triaging a faulted deployment and deciding between fixing forward and rolling back, within a defined time window.

When to Use This Runbook

• A deployment completes but the service fails its post-deploy health check.
• Error rates or latency spike immediately following a deployment.
• A deployment is flagged as faulted by the CI/CD pipeline or monitoring.

SLA

Milestone	Target
Acknowledge faulted deployment	5 minutes
Triage window	30 minutes from acknowledgment
Decision: fix forward or roll back	At the 30-minute mark, no later

If the root cause has not been identified and confirmed fixable within the 30-minute triage window, the deployment is rolled back. Fixing forward past the 30-minute mark requires sign-off from the service owner.

Triage Steps (0–30 Minutes)

1. Confirm the fault: review the failed health check, error logs, and any CI/CD pipeline warnings.
2. Loop in QA to pinpoint the specific failures introduced by the deployment, testing affected workflows to isolate what broke.
3. Identify the likely cause: configuration error, missing dependency, code defect, or environment mismatch.
4. Assess whether the cause is understood and a fix can be applied and verified within the remaining triage window.
5. If yes, proceed to Fix Forward. If no, or time expires, proceed to Rollback.

Fix Forward Path

1. Obtain sign-off from the service owner to proceed past the triage window if applicable.
2. Apply the fix and redeploy.
3. Re-run the post-deploy health check to confirm resolution.
4. If the fix does not resolve the issue, immediately proceed to Rollback rather than continuing to troubleshoot in production.

Rollback Path

1. Initiate rollback to the last known-good version: deploy rollback --service order-processing-api --to-version [last-known-good].
2. Confirm rollback completion in the CI/CD dashboard.
3. Re-run the post-deploy health check to confirm the service has returned to a healthy state.
4. Notify the deployment owner and on-call channel that the rollback is complete.

Escalation Path

1. On-call SRE (first responder)
2. QA (pinpoints failures introduced by the deployment during triage)
3. Deployment owner / engineer who initiated the deploy
4. Service owner (required sign-off past the 30-minute triage window)
5. Incident commander (contacted for any faulted deployment, regardless of scope, and briefed on the identified cause once known, unless the issue is confirmed to be a simple bug)

Post-Incident

• Log the fault, triage timeline, and decision (fix forward vs. rollback) in the incident tracker.
• The on-call SRE submits a Root Cause Analysis (RCA) for approval by the team leader within 5 days of the incident. The RCA includes the incident timeline, resolution, and suggestions for change.
• A corrective actions document is created to document and track any necessary long-term changes identified by the RCA.
• If rolled back, schedule a follow-up review before the fix is reattempted.
• Feed recurring fault causes back into the change management / go-no-go review criteria.

Voice & Tone

• Use imperative, instructional voice (“Restart the service”); avoid second person (“you”) and first person (“I”, “we”) in technical writing.
• Use active voice: “Restart the service” rather than “The service should be restarted.”
• Keep sentences short and direct. One instruction per step.
• Avoid humor, idioms, or culturally specific references. Documentation should translate cleanly across onshore and offshore readers.

Formatting Conventions

• Use numbered lists for sequential steps; use bulleted lists for non-sequential items with more than three points. Fewer than three items can stay in-line as a sentence.
• Use “select” instead of “click” when referring to UI actions (e.g., select Save Changes), since not all interactions are mouse clicks.
• Use code formatting for commands, file paths, and variable names.
• Acronyms must be spelled out on first mention, followed by the acronym in parentheses (e.g., Root Cause Analysis (RCA)). The acronym may be used on its own thereafter.
• Every runbook and playbook must include: Purpose, When to Use, Severity/Priority (if applicable), Steps, and Escalation Path, in that order.

Terminology

Use	Avoid
Incident	Issue, problem (in incident docs)
Rollback	Revert, undo
On-call engineer	Responder, the engineer on duty
Service owner	Point of contact, owner

Maintain a living glossary for any team-specific or product-specific terms. New terms are approved by the documentation lead before use in published content.

Review & Approval

• All runbooks and playbooks require a technical review from the owning engineering team before publishing.
• Style and formatting review is performed by the documentation team regardless of technical review outcome.
• Published documents are reviewed on a quarterly cadence to confirm accuracy.