영어 인시던트 커뮤니케이션 완전 가이드: 장애 보고부터 포스트모템 작성까지 실전 표현

# SEV1 선언
"I'm declaring this a SEV1 incident. We have a complete outage of our payment processing service."
(이것을 SEV1 인시던트로 선언합니다. 결제 처리 서비스가 완전히 중단되었습니다.)

# SEV2 선언
"This is being escalated to SEV2. We're seeing significant degradation across multiple services."
(SEV2로 에스컬레이션합니다. 여러 서비스에서 심각한 성능 저하가 관찰됩니다.)

# 심각도 변경 (상향)
"We're upgrading this from SEV3 to SEV2. The blast radius is wider than initially assessed."
(SEV3에서 SEV2로 상향 조정합니다. 영향 범위가 최초 평가보다 넓습니다.)

# 심각도 변경 (하향)
"Based on the reduced impact, we're downgrading this to SEV3."
(영향도 감소에 따라 SEV3으로 하향 조정합니다.)

# 장애 감지 보고
"We've detected an anomaly in [서비스명]. [증상 설명]."
(~에서 이상 징후를 감지했습니다.)

"Our monitoring has triggered alerts for [서비스명]. We're seeing [증상]."
(모니터링이 ~에 대한 알림을 발생시켰습니다. ~이 관찰됩니다.)

"We're currently investigating elevated error rates on [서비스명]."
(현재 ~의 높아진 에러율을 조사 중입니다.)

# 영향 범위 보고
"Impact: Approximately [X]% of users are unable to [기능]."
(영향: 약 X%의 사용자가 ~을 할 수 없습니다.)

"This is affecting all users in the [지역] region."
(~지역의 모든 사용자에게 영향을 미치고 있습니다.)

"The blast radius includes [서비스 A], [서비스 B], and [서비스 C]."
(영향 범위에 ~, ~, ~이 포함됩니다.)

# 현재 조치 보고
"We have identified the issue and are working on a fix."
(문제를 확인했으며 수정 작업 중입니다.)

"We're actively investigating. No root cause identified yet."
(적극적으로 조사 중입니다. 아직 근본 원인은 파악되지 않았습니다.)

"A rollback has been initiated and we expect resolution within [시간]."
(롤백이 시작되었으며 ~이내 해결될 것으로 예상합니다.)

Subject: [SEV1] Payment Service Outage - Incident #INC-2026-0308

Team,

Incident Summary:
- Incident ID: INC-2026-0308
- Severity: SEV1 (Critical)
- Detected at: 2026-03-08 03:15 UTC
- Incident Commander: Jane Smith
- Status: Investigating

What happened:
At approximately 03:15 UTC, our monitoring detected a complete failure
of the payment processing pipeline. All transaction attempts are
returning 500 errors.

Impact:
- 100% of payment transactions are failing
- Estimated revenue impact: ~$50K/hour
- All regions affected
- Approximately 15,000 users impacted

Current Actions:
- On-call SRE team has been paged and is actively investigating
- Database team has been engaged
- We are examining recent deployment changes as a potential cause

Next Update:
We will provide an update within 30 minutes or sooner if we identify
the root cause.

Incident Channel: #inc-2026-0308-payment-outage

Regards,
[이름]
Incident Commander

# 기술적 에스컬레이션 (다른 팀 도움 요청)
"We need to escalate this to the database team. The issue appears to
be related to connection pool exhaustion."
(이 건을 데이터베이스 팀으로 에스컬레이션해야 합니다. 커넥션 풀 고갈과
관련된 문제로 보입니다.)

# 매니지먼트 에스컬레이션 (경영진 보고)
"I'm escalating this to VP-level. The customer impact has exceeded
our SEV1 threshold and we may need to issue a public statement."
(VP 레벨로 에스컬레이션합니다. 고객 영향이 SEV1 기준을 초과했으며
공식 성명 발표가 필요할 수 있습니다.)

# 벤더 에스컬레이션
"We've exhausted our internal troubleshooting options. I'm opening a
Priority 1 support case with AWS/GCP."
(내부 트러블슈팅 옵션을 모두 소진했습니다. AWS/GCP에 Priority 1
지원 케이스를 오픈합니다.)

# 인수인계 (Handoff)
"I've been on-call for 6 hours and need to hand off IC duties.
Here's the current status summary for the incoming IC."
(6시간 동안 온콜 중이었으며 IC 업무를 인수인계해야 합니다.
들어오는 IC를 위한 현재 상태 요약입니다.)

# 조사 중
"We're still investigating. We've narrowed down the issue to the
caching layer but haven't identified the root cause yet."
(계속 조사 중입니다. 캐싱 레이어로 문제를 좁혔지만 아직 근본 원인은
파악하지 못했습니다.)

# 원인 파악
"Root cause identified: A misconfigured rate limiter is dropping
legitimate traffic. Working on a fix now."
(근본 원인 파악: 잘못 설정된 레이트 리미터가 정상 트래픽을 차단하고
있습니다. 현재 수정 작업 중입니다.)

# 부분 복구
"Partial mitigation in place. We've rerouted traffic to healthy
instances. 80% of users should now have normal access."
(부분 완화 조치 적용 중. 정상 인스턴스로 트래픽을 재라우팅했습니다.
80%의 사용자가 정상 접근 가능합니다.)

# 완전 복구
"The incident has been fully resolved. All services are operating
normally. We will continue monitoring for the next 2 hours."
(인시던트가 완전히 해결되었습니다. 모든 서비스가 정상 운영 중입니다.
향후 2시간 동안 모니터링을 계속합니다.)

# 모니터링 중
"Fix has been deployed. We're monitoring metrics closely.
No recurrence observed so far."
(수정이 배포되었습니다. 메트릭을 면밀히 모니터링 중입니다.
현재까지 재발은 관찰되지 않았습니다.)

# 타임라인 기록 예시
03:15 UTC - Monitoring alert triggered for payment-service error rate spike
03:17 UTC - On-call engineer acknowledged the alert
03:22 UTC - Incident declared as SEV1, war room opened
03:30 UTC - Root cause identified: bad config push in payment-gateway
03:35 UTC - Rollback initiated for payment-gateway config
03:42 UTC - Rollback completed, error rates returning to normal
03:55 UTC - All metrics back to baseline, incident marked as resolved
04:00 UTC - Post-incident monitoring period started (2 hours)

:rotating_light: INCIDENT DECLARED :rotating_light:

Incident: INC-2026-0308
Severity: SEV1 - Critical
Title: Payment Service Complete Outage

Roles:
- Incident Commander (IC): @jane.smith
- Communications Lead: @bob.lee
- Operations Lead: @alex.chen

Impact: All payment transactions failing globally
Status: Investigating

Ground rules:
- Keep this channel for incident-related discussion only
- Use threads for detailed technical discussions
- IC will post regular status updates every 15 minutes

Next update: 03:45 UTC

# 상태 업데이트 (진행 중)
:mag: UPDATE (03:45 UTC)
Status: Still investigating
- Confirmed: Issue is NOT related to the 02:00 UTC deployment
- Currently examining database connection pool metrics
- @db-team has been engaged
Next update: 04:00 UTC

# 원인 파악 시
:bulb: ROOT CAUSE IDENTIFIED (04:00 UTC)
The connection pool max size was inadvertently reduced from 100 to 10
in yesterday's config change (PR #4521).
Action: Reverting config change now. ETA for resolution: 15 minutes.

# 완화 조치 적용 시
:yellow_circle: PARTIAL MITIGATION (04:10 UTC)
Config has been reverted. Error rates dropping.
- Payment success rate: 45% -> 92% (recovering)
- Still monitoring for full recovery
- Some queued transactions may need manual processing

# 해결 시
:white_check_mark: RESOLVED (04:25 UTC)
All services fully restored. Payment success rate back to 99.9%.
- Duration: 1 hour 10 minutes
- Root cause: Config change reducing DB connection pool
- Postmortem will be scheduled within 48 hours
- Monitoring will continue for the next 2 hours

Thank you everyone for the quick response.

=== INVESTIGATING ===
Title: Degraded Performance on Payment Processing

[Posted: March 8, 2026 03:20 UTC]
We are currently investigating reports of failed payment transactions.
Some users may experience errors when attempting to complete purchases.
Our engineering team is actively working to identify and resolve the
issue. We will provide an update within 30 minutes.

No data loss or security concerns have been identified at this time.


=== IDENTIFIED ===
Title: Payment Processing - Root Cause Identified

[Updated: March 8, 2026 04:00 UTC]
We have identified the root cause of the payment processing issues.
A configuration change is being applied to restore normal service.
We expect full resolution within the next 15-20 minutes.

Affected services: Payment processing, order confirmation emails
Unaffected services: User accounts, product browsing, search


=== MONITORING ===
Title: Payment Processing - Fix Applied, Monitoring

[Updated: March 8, 2026 04:15 UTC]
A fix has been applied and payment processing is recovering.
Most users should now be able to complete transactions normally.
We are closely monitoring the situation to ensure stability.

If you experienced a failed transaction, please retry. If the issue
persists, contact support@example.com.


=== RESOLVED ===
Title: Payment Processing - Fully Resolved

[Updated: March 8, 2026 04:30 UTC]
The payment processing issue has been fully resolved. All services
are operating normally.

Duration: Approximately 1 hour 15 minutes (03:15 - 04:30 UTC)

We sincerely apologize for any inconvenience this may have caused.
A detailed analysis of this incident will be conducted to prevent
recurrence.

For any ongoing concerns, please contact support@example.com.

# 비난하는 표현 (피해야 할 표현)
"John made a mistake and pushed the wrong config."
(John이 실수해서 잘못된 설정을 배포했다.)

# Blameless 표현 (사용해야 할 표현)
"The config deployment process lacked sufficient validation checks,
which allowed an incorrect configuration to reach production."
(설정 배포 프로세스에 충분한 검증 단계가 부족하여, 잘못된 설정이
프로덕션까지 도달할 수 있었다.)

# Postmortem: Payment Service Outage

## Incident ID: INC-2026-0308

### Metadata

| Field              | Value                 |
| ------------------ | --------------------- |
| Date               | March 8, 2026         |
| Authors            | Jane Smith, Alex Chen |
| Status             | Complete              |
| Severity           | SEV1                  |
| Duration           | 1 hour 10 minutes     |
| Incident Commander | Jane Smith            |

---

### Executive Summary

On March 8, 2026, from 03:15 to 04:25 UTC, users experienced a
complete inability to process payments. The root cause was a
configuration change that reduced the database connection pool
from 100 to 10 connections, causing connection exhaustion under
normal load. Approximately 15,000 users were affected, with an
estimated revenue impact of ~$60,000.

---

### Impact

- **Duration**: 1 hour 10 minutes (03:15 - 04:25 UTC)
- **Users affected**: ~15,000 (100% of payment attempts)
- **Revenue impact**: ~$60,000 estimated
- **Support tickets**: 342 tickets opened
- **SLA impact**: Monthly uptime dropped from 99.95% to 99.87%

---

### Timeline (all times UTC)

| Time  | Event                                                      |
| ----- | ---------------------------------------------------------- |
| 02:00 | Config change deployed via PR #4521 (unrelated to payment) |
| 03:15 | Payment error rate alert triggered (threshold: >1%)        |
| 03:17 | On-call SRE acknowledged alert                             |
| 03:22 | SEV1 declared, incident channel created                    |
| 03:25 | Initial status page update posted                          |
| 03:30 | DB team engaged, connection pool exhaustion identified     |
| 03:35 | Root cause confirmed: PR #4521 reduced pool size           |
| 03:38 | Config rollback initiated                                  |
| 03:42 | Rollback completed                                         |
| 03:55 | Error rates returned to baseline                           |
| 04:00 | Status page updated: monitoring                            |
| 04:25 | Incident resolved, all metrics nominal for 30 minutes      |

---

### Root Cause

PR #4521, intended to update logging configuration, inadvertently
modified the database connection pool size parameter from 100 to 10.
The change passed code review because the pool size parameter was
in the same configuration file as logging settings. Under normal
traffic load (~80 concurrent connections), the reduced pool caused
connection exhaustion, resulting in 500 errors for all payment
requests.

---

### What Went Well

- Alert fired within 2 minutes of impact starting
- On-call engineer responded within 2 minutes
- Incident was declared and war room opened within 7 minutes
- Clear communication maintained throughout

### What Went Poorly

- Config change was not caught in code review
- No automated validation for critical infrastructure parameters
- Rollback took 7 minutes due to manual approval process
- Payment service had no circuit breaker for DB connections

### Where We Got Lucky

- The incident occurred during low-traffic hours (UTC night)
- No data corruption occurred despite connection failures
- A team member happened to be online and recognized the
  config change immediately

---

### Action Items

| ID  | Action                                                  | Type     | Owner | Priority | Due Date |
| --- | ------------------------------------------------------- | -------- | ----- | -------- | -------- |
| 1   | Separate infrastructure config from application config  | Prevent  | @alex | P1       | March 15 |
| 2   | Add automated validation for connection pool parameters | Prevent  | @alex | P1       | March 15 |
| 3   | Implement circuit breaker pattern for DB connections    | Mitigate | @bob  | P2       | March 22 |
| 4   | Add connection pool size to deployment canary checks    | Detect   | @jane | P1       | March 18 |
| 5   | Automate rollback process for config changes            | Mitigate | @ops  | P2       | March 29 |
| 6   | Add runbook for payment service connection failures     | Process  | @jane | P3       | April 5  |

---

### Lessons Learned

Infrastructure-critical parameters should be managed separately
from application-level configuration and require additional
validation gates before deployment.

# 원인 설명
"The root cause was determined to be..."
(근본 원인은 ~으로 판명되었습니다.)

"Contributing factors included..."
(기여 요인에는 ~이 포함됩니다.)

"The issue was exacerbated by..."
(문제가 ~에 의해 악화되었습니다.)

# 영향 설명
"This resulted in a total of [X] minutes of downtime."
(이로 인해 총 X분의 다운타임이 발생했습니다.)

"The blast radius was limited to [서비스/지역]."
(영향 범위는 ~로 제한되었습니다.)

"User-facing impact lasted approximately [X] minutes."
(사용자 대면 영향은 약 X분간 지속되었습니다.)

# 액션 아이템 설명
"To prevent recurrence, we will implement..."
(재발 방지를 위해 ~을 구현할 것입니다.)

"As a short-term mitigation, we have..."
(단기 완화 조치로 ~했습니다.)

"Long-term, we plan to redesign the [시스템] to..."
(장기적으로 ~를 재설계할 계획입니다.)

# 5 Whys Analysis - Payment Service Outage (INC-2026-0308)

Problem Statement:
Payment service experienced a complete outage for 70 minutes.
(결제 서비스가 70분간 완전 중단되었습니다.)

Why 1: Why did the payment service go down?
-> Database connection pool was exhausted, causing all payment
   requests to fail with timeout errors.
(DB 커넥션 풀이 고갈되어 모든 결제 요청이 타임아웃 에러로 실패했습니다.)

Why 2: Why was the connection pool exhausted?
-> The maximum pool size was reduced from 100 to 10, which is
   insufficient for normal production traffic.
(최대 풀 크기가 100에서 10으로 줄어들어 정상 프로덕션 트래픽에
불충분했습니다.)

Why 3: Why was the pool size reduced?
-> A configuration change in PR #4521 unintentionally modified the
   pool size parameter alongside logging configuration changes.
(PR #4521의 설정 변경이 로깅 설정 변경과 함께 의도치 않게
풀 크기 파라미터를 수정했습니다.)

Why 4: Why wasn't this caught during code review?
-> The pool size parameter was co-located with logging config in
   a single file, making it easy to overlook. The reviewer focused
   on the logging changes and missed the pool size modification.
(풀 크기 파라미터가 로깅 설정과 같은 파일에 있어 간과하기 쉬웠습니다.
리뷰어가 로깅 변경에 집중하여 풀 크기 수정을 놓쳤습니다.)

Why 5: Why was there no automated validation for critical parameters?
-> Our CI/CD pipeline does not include infrastructure parameter
   validation. Configuration changes follow the same review process
   as application code without additional safeguards.
(CI/CD 파이프라인에 인프라 파라미터 검증이 포함되어 있지 않습니다.
설정 변경이 추가 안전장치 없이 애플리케이션 코드와 동일한 리뷰
프로세스를 따릅니다.)

Root Cause:
Lack of separation between infrastructure-critical configuration
and application-level configuration, combined with absence of
automated validation for production-critical parameters.

(인프라 핵심 설정과 애플리케이션 수준 설정 간의 분리 부족, 그리고
프로덕션 핵심 파라미터에 대한 자동 검증 부재.)

# 미팅 오프닝
"Thank you all for joining this postmortem review for INC-2026-0308.
Before we begin, I want to emphasize that this is a blameless
postmortem. We're here to learn from the incident and improve our
systems, not to assign blame."
(INC-2026-0308 포스트모템 리뷰에 참석해 주셔서 감사합니다. 시작하기
전에, 이것은 비난 없는 포스트모템임을 강조하고 싶습니다. 우리는 인시던트
로부터 배우고 시스템을 개선하기 위해 모인 것이지, 비난하기 위함이
아닙니다.)

# 배경 설명
"Let me walk you through the timeline of events."
(사건 타임라인을 함께 살펴보겠습니다.)

"Here's what we know about the sequence of events."
(사건 경과에 대해 파악된 내용입니다.)

# 추가 정보 요청
"Can you elaborate on what you observed at that point?"
(그 시점에서 관찰한 내용을 더 자세히 설명해 주시겠어요?)

"What information did you have available when you made that decision?"
(그 결정을 내릴 때 어떤 정보를 갖고 계셨나요?)

# 개선점 논의
"What could we have done differently to detect this sooner?"
(이것을 더 빨리 감지하기 위해 무엇을 다르게 할 수 있었을까요?)

"Are there any systemic improvements we should consider?"
(고려해야 할 시스템적 개선 사항이 있을까요?)

"What guardrails can we put in place to prevent this class of issues?"
(이 유형의 문제를 방지하기 위해 어떤 안전장치를 마련할 수 있을까요?)

# 액션 아이템 확정
"Let's make sure each action item has a clear owner and due date."
(각 액션 아이템에 명확한 담당자와 마감일이 있는지 확인합시다.)

"Who would like to own this action item?"
(이 액션 아이템을 담당하실 분이 계신가요?)

# 요약 및 마무리
"To summarize, we've identified [X] action items. Let me read them
back to confirm alignment."
(요약하면, X개의 액션 아이템을 도출했습니다. 합의를 확인하기 위해
다시 읽어 보겠습니다.)

"Does anyone have any additional observations or concerns before
we close out this postmortem?"
(포스트모템을 마무리하기 전에 추가 의견이나 우려 사항이 있으신 분
계신가요?)

"The postmortem document will be shared in [위치] by end of day.
Please review and add any comments within the next 48 hours."
(포스트모템 문서는 오늘 중으로 ~에 공유됩니다.
48시간 이내에 검토 후 코멘트를 추가해 주세요.)

# 너무 캐주얼한 톤 (피해야 할 표현)
"Hey guys, looks like something broke in prod. Looking into it."

# 적절한 전문적 톤 (사용해야 할 표현)
"Team, we're investigating an issue affecting production services.
Will provide an update within 15 minutes."

# 너무 기술적인 톤 (고객 커뮤니케이션에서 피해야 할 표현)
"A null pointer exception in the payment microservice's gRPC handler
caused a cascading failure across the service mesh."

# 고객에게 적절한 톤
"We identified a technical issue in our payment processing system
that caused some transactions to fail. The issue has been resolved."

INCIDENT COMMUNICATION CHECKLIST
=================================

[ ] DETECTION (감지 단계)
    - Acknowledge the alert within 5 minutes
      (5분 이내 알림 확인)
    - Assess initial severity level
      (초기 심각도 수준 평가)
    - Create incident channel (if SEV1/SEV2)
      (인시던트 채널 생성 - SEV1/SEV2인 경우)

[ ] INITIAL RESPONSE (초기 대응)
    - Declare the incident and assign roles
      (인시던트 선언 및 역할 배정)
      - Incident Commander (IC)
      - Communications Lead
      - Operations Lead
    - Send initial notification within 5 minutes
      (5분 이내 초기 알림 발송)
    - Post first status page update
      (첫 상태 페이지 업데이트 게시)

[ ] DURING INCIDENT (인시던트 진행 중)
    - Provide updates every 15-20 minutes
      (15-20분마다 업데이트 제공)
    - Update status page at each phase change
      (각 단계 변경 시 상태 페이지 업데이트)
    - Escalate if needed (technical or management)
      (필요 시 에스컬레이션)
    - Document all actions in the timeline
      (모든 조치를 타임라인에 기록)

[ ] RESOLUTION (해결)
    - Confirm all services restored
      (모든 서비스 복구 확인)
    - Post final status page update
      (최종 상태 페이지 업데이트 게시)
    - Send resolution notification to stakeholders
      (이해관계자에게 해결 알림 발송)
    - Begin post-incident monitoring period
      (인시던트 후 모니터링 기간 시작)

[ ] POST-INCIDENT (인시던트 후)
    - Schedule postmortem within 48 hours
      (48시간 이내 포스트모템 일정 잡기)
    - Complete postmortem document
      (포스트모템 문서 완성)
    - Conduct postmortem review meeting
      (포스트모템 리뷰 미팅 진행)
    - Track action items to completion
      (액션 아이템 완료까지 추적)
    - Share lessons learned broadly
      (학습한 교훈을 널리 공유)