How Is Large-Scale Software Developed, Managed, and Maintained? — Including SW Update Systems

company-repo/
  frontend/
  backend/
  mobile/
  infra/
  shared-libs/
  tools/

# API Gateway routing example
routes:
  - path: /api/users
    service: user-service
    methods: [GET, POST]
  - path: /api/orders
    service: order-service
    methods: [GET, POST, PUT]
  - path: /api/payments
    service: payment-service
    methods: [POST]

# Istio VirtualService example
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: user-service
spec:
  hosts:
    - user-service
  http:
    - route:
        - destination:
            host: user-service
            subset: v2
          weight: 90
        - destination:
            host: user-service
            subset: v1
          weight: 10

# Circuit Breaker pseudocode
class CircuitBreaker:
    def __init__(self, threshold=5, timeout=30):
        self.failure_count = 0
        self.threshold = threshold
        self.timeout = timeout
        self.state = "CLOSED"  # CLOSED, OPEN, HALF_OPEN

    def call(self, func):
        if self.state == "OPEN":
            if self.timeout_expired():
                self.state = "HALF_OPEN"
            else:
                raise CircuitOpenError("Service unavailable")

        try:
            result = func()
            self.on_success()
            return result
        except Exception as e:
            self.on_failure()
            raise e

    def on_failure(self):
        self.failure_count += 1
        if self.failure_count >= self.threshold:
            self.state = "OPEN"

    def on_success(self):
        self.failure_count = 0
        self.state = "CLOSED"

# GitHub Actions CI Pipeline example
name: CI Pipeline
on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
      - name: Install dependencies
        run: npm ci
      - name: Lint
        run: npm run lint
      - name: Unit Tests
        run: npm run test:unit
      - name: Integration Tests
        run: npm run test:integration
      - name: Build
        run: npm run build

Traffic distribution:
  v1 (current): 95%  ████████████████████░
  v2 (new):      5%  █░░░░░░░░░░░░░░░░░░░

After 30 min (no issues):
  v1 (current): 50%  ██████████░░░░░░░░░░
  v2 (new):     50%  ██████████░░░░░░░░░░

After 1 hour (no issues):
  v1 (current):  0%  ░░░░░░░░░░░░░░░░░░░░
  v2 (new):    100%  ████████████████████

# Kubernetes Rolling Update configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    spec:
      containers:
        - name: my-app
          image: my-app:v2
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10

// Feature flag usage example
const featureFlags = {
  newCheckoutFlow: false,
  darkMode: true,
  aiRecommendation: false,
};

function renderCheckout() {
  if (featureFlags.newCheckoutFlow) {
    return renderNewCheckout();
  }
  return renderLegacyCheckout();
}

// Electron auto-updater implementation
import { autoUpdater } from 'electron-updater';

autoUpdater.setFeedURL({
  provider: 'github',
  owner: 'my-org',
  repo: 'my-app',
});

autoUpdater.on('update-available', (info) => {
  log.info('New version found:', info.version);
  showUpdateNotification(info);
});

autoUpdater.on('download-progress', (progress) => {
  updateProgressBar(progress.percent);
});

autoUpdater.on('update-downloaded', (info) => {
  showRestartDialog(info.version);
});

// Check for updates every 4 hours
setInterval(() => {
  autoUpdater.checkForUpdates();
}, 4 * 60 * 60 * 1000);

Full update:  v1.0 (200MB) -> v1.1 (200MB) = 200MB download
Delta update: v1.0 (200MB) -> v1.1 (200MB) = 15MB download (diff only)

Day 1: 1% rollout -> Monitor crash reports
Day 2: 5% rollout -> Check user feedback
Day 3: 20% rollout
Day 5: 50% rollout
Day 7: 100% rollout

// React Native OTA Update (EAS Update example)
import * as Updates from 'expo-updates';

async function checkForOTAUpdate() {
  try {
    const update = await Updates.checkForUpdateAsync();
    if (update.isAvailable) {
      await Updates.fetchUpdateAsync();
      Alert.alert(
        'Update Complete',
        'A new version is ready. Apply now?',
        [
          { text: 'Later', style: 'cancel' },
          { text: 'Apply Now', onPress: () => Updates.reloadAsync() },
        ]
      );
    }
  } catch (error) {
    console.error('OTA update failed:', error);
  }
}

# ArgoCD GitOps-based server update
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-service
spec:
  project: default
  source:
    repoURL: https://github.com/my-org/k8s-manifests
    path: services/my-service
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Deliberate + Prudent: "Let's go with a simple implementation now and refactor next quarter"
Deliberate + Reckless: "Skip the design, just build it fast"
Inadvertent + Prudent: "Looking back after implementation, a better design existed"
Inadvertent + Reckless: "What's a layered architecture?"

Phase 1: Legacy(100%) --- New(0%)
Phase 2: Legacy(70%)  --- New(30%)
Phase 3: Legacy(30%)  --- New(70%)
Phase 4: Legacy(0%)   --- New(100%)  <- Remove legacy

# Postmortem Template

## Incident Summary
- Occurred: 2026-04-10 14:23 KST
- Recovered: 2026-04-10 15:07 KST
- Impact: ~30% of all users (payment service)
- Severity: P1 (Critical)

## Timeline
- 14:23 - Payment success rate drop alert triggered
- 14:25 - On-call engineer response started
- 14:32 - Root cause identified (DB connection pool exhaustion)
- 14:45 - Emergency patch deployment started
- 15:07 - Full service recovery confirmed

## Root Cause
A newly deployed query was missing an index, increasing response time
from 50ms to 5s, causing connection pool exhaustion.

## Action Items
- [Done] Added missing index
- [In Progress] Build automated slow query detection
- [Planned] Automate query performance testing before deployment

SLO: 99.95% availability (monthly)

This month: 99.97%
Error budget remaining: 0.02%
Status: New feature deployment OK

If 99.93%?
Error budget: Exhausted
Status: Focus on stability only

Traditional corporate development:
  Team A -> Request Library X -> Team B -> Backlog wait (2-3 months)

InnerSource:
  Team A -> Submit PR to Library X -> Team B review -> Merge (1-2 weeks)

Architecture
  [ ] Are service boundaries clearly defined?
  [ ] Is there an API versioning strategy?
  [ ] Are circuit breakers / retry policies implemented?

CI/CD
  [ ] Is the path from code commit to production automated?
  [ ] Is rollback possible within 5 minutes?
  [ ] Is there a canary/blue-green deployment strategy?

Update System
  [ ] Is auto-update implemented?
  [ ] Is update file integrity verified (checksum, signature)?
  [ ] Is there a rollback mechanism for update failures?

Monitoring
  [ ] Are SLI/SLOs defined?
  [ ] Is there an alert escalation policy?
  [ ] Is distributed tracing implemented?

Technical Debt
  [ ] Is technical debt measured regularly?
  [ ] Is engineering time allocated for refactoring?
  [ ] Is there a dependency update policy?

Documentation
  [ ] Are ADRs being written?
  [ ] Are API docs auto-generated?
  [ ] Is onboarding documentation up to date?