Cohere Forward Deployed Engineer 합격 가이드: AI 플랫폼 배포 전문가가 되는 완벽 로드맵

                     ┌─────────────────────────────┐
                     │       Control Plane          │
                     │  ┌─────────┐ ┌───────────┐  │
                     │  │kube-api │ │ scheduler │  │
                     │  │ server  │ │           │  │
                     │  └────┬────┘ └───────────┘  │
                     │  ┌────┴────┐ ┌───────────┐  │
                     │  │  etcd   │ │controller │  │
                     │  │         │ │ manager   │  │
                     │  └─────────┘ └───────────┘  │
                     └──────────────┬──────────────┘
                                    │
              ┌─────────────────────┼─────────────────────┐
              │                     │                     │
     ┌────────┴────────┐  ┌────────┴────────┐  ┌────────┴────────┐
     │   Worker Node 1  │  │   Worker Node 2  │  │  GPU Node (AI)  │
     │  ┌─────┐┌─────┐ │  │  ┌─────┐┌─────┐ │  │  ┌─────┐┌─────┐│
     │  │ Pod ││ Pod │ │  │  │ Pod ││ Pod │ │  │  │ Pod ││ Pod ││
     │  └─────┘└─────┘ │  │  └─────┘└─────┘ │  │  │ GPU ││ GPU ││
     │  kubelet+kproxy  │  │  kubelet+kproxy  │  │  └─────┘└─────┘│
     └──────────────────┘  └──────────────────┘  └─────────────────┘

# Rolling Update - 가장 일반적
apiVersion: apps/v1
kind: Deployment
metadata:
  name: north-ai-api
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    spec:
      containers:
        - name: north-api
          image: cohere/north-api:v2.1.0
          resources:
            requests:
              memory: '4Gi'
              cpu: '2'
            limits:
              memory: '8Gi'
              cpu: '4'
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 30

# 고객별 네임스페이스 격리
apiVersion: v1
kind: Namespace
metadata:
  name: customer-rbc
  labels:
    customer: rbc
    environment: production
---
# 최소 권한 원칙에 따른 Role 정의
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: customer-rbc
  name: north-deployer
rules:
  - apiGroups: ['apps']
    resources: ['deployments', 'statefulsets']
    verbs: ['get', 'list', 'watch', 'create', 'update', 'patch']
  - apiGroups: ['']
    resources: ['pods', 'services', 'configmaps', 'secrets']
    verbs: ['get', 'list', 'watch', 'create', 'update']
  - apiGroups: ['']
    resources: ['pods/log']
    verbs: ['get']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: customer-rbc
  name: north-deployer-binding
subjects:
  - kind: ServiceAccount
    name: north-deploy-sa
    namespace: customer-rbc
roleRef:
  kind: Role
  name: north-deployer
  apiGroup: rbac.authorization.k8s.io

# North AI 플랫폼 Pod 간 통신만 허용
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: north-platform-policy
  namespace: customer-rbc
spec:
  podSelector:
    matchLabels:
      app: north-platform
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: north-platform
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: north-platform
      ports:
        - protocol: TCP
          port: 8080
    - to: # DNS 허용
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53

# LimitRange로 네임스페이스 기본값 설정
apiVersion: v1
kind: LimitRange
metadata:
  name: north-limits
  namespace: customer-rbc
spec:
  limits:
    - default:
        memory: '2Gi'
        cpu: '1'
      defaultRequest:
        memory: '512Mi'
        cpu: '250m'
      type: Container
---
# ResourceQuota로 네임스페이스 전체 제한
apiVersion: v1
kind: ResourceQuota
metadata:
  name: north-quota
  namespace: customer-rbc
spec:
  hard:
    requests.cpu: '32'
    requests.memory: '64Gi'
    limits.cpu: '64'
    limits.memory: '128Gi'
    requests.nvidia.com/gpu: '8'
    pods: '50'

# GPU 노드에 Pod 배치를 위한 tolerations + nodeSelector
apiVersion: apps/v1
kind: Deployment
metadata:
  name: north-model-server
spec:
  template:
    spec:
      nodeSelector:
        accelerator: nvidia-a100
      tolerations:
        - key: 'nvidia.com/gpu'
          operator: 'Exists'
          effect: 'NoSchedule'
      containers:
        - name: model-server
          image: cohere/north-model:latest
          resources:
            limits:
              nvidia.com/gpu: 4
          volumeMounts:
            - name: model-storage
              mountPath: /models
            - name: shm
              mountPath: /dev/shm
      volumes:
        - name: model-storage
          persistentVolumeClaim:
            claimName: model-pvc
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: '16Gi'

# Prometheus ServiceMonitor 설정
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: north-platform-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: north-platform
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

# 1. 문제 노드 식별
kubectl get nodes -o wide
kubectl describe node problematic-node

# 2. 노드 cordon (새 Pod 스케줄링 차단)
kubectl cordon problematic-node

# 3. 기존 워크로드 안전하게 이동 (drain)
kubectl drain problematic-node --ignore-daemonsets --delete-emptydir-data

# 4. 문제 해결 후 노드 복구
kubectl uncordon problematic-node

# 5. Pod 상태 확인
kubectl get pods -n customer-rbc -o wide
kubectl logs -n customer-rbc pod-name --previous

north-ai-platform/
  Chart.yaml          # 차트 메타데이터
  Chart.lock          # 의존성 잠금 파일
  values.yaml         # 기본 설정값
  values-production.yaml   # 프로덕션 오버라이드
  values-staging.yaml      # 스테이징 오버라이드
  templates/
    _helpers.tpl      # 공통 템플릿 함수
    deployment.yaml   # Deployment 리소스
    service.yaml      # Service 리소스
    ingress.yaml      # Ingress 리소스
    configmap.yaml    # ConfigMap
    secret.yaml       # Secret
    hpa.yaml          # HorizontalPodAutoscaler
    pdb.yaml          # PodDisruptionBudget
    networkpolicy.yaml
    serviceaccount.yaml
    NOTES.txt         # 설치 후 안내 메시지
  charts/             # 하위 차트 (의존성)
  tests/
    test-connection.yaml

apiVersion: v2
name: north-ai-platform
description: Cohere North AI Platform for Enterprise Deployment
type: application
version: 2.1.0 # Chart 버전
appVersion: '3.5.0' # 애플리케이션 버전
keywords:
  - ai
  - llm
  - cohere
  - enterprise
dependencies:
  - name: postgresql
    version: '12.x.x'
    repository: 'https://charts.bitnami.com/bitnami'
    condition: postgresql.enabled
  - name: redis
    version: '17.x.x'
    repository: 'https://charts.bitnami.com/bitnami'
    condition: redis.enabled

# templates/_helpers.tpl
{{- define "north.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}

{{- define "north.labels" -}}
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" }}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{- define "north.gpuResources" -}}
{{- if .Values.gpu.enabled }}
resources:
  limits:
    nvidia.com/gpu: {{ .Values.gpu.count | default 1 }}
  requests:
    nvidia.com/gpu: {{ .Values.gpu.count | default 1 }}
{{- end }}
{{- end }}

# values.yaml (기본값)
replicaCount: 1
image:
  repository: cohere/north-platform
  tag: 'latest'
  pullPolicy: IfNotPresent

modelServer:
  replicas: 1
  gpu:
    enabled: false
    count: 1
    type: ''
  resources:
    requests:
      memory: '8Gi'
      cpu: '4'
    limits:
      memory: '16Gi'
      cpu: '8'

ingress:
  enabled: true
  className: nginx
  tls:
    enabled: true

monitoring:
  enabled: true
  prometheus:
    scrape: true

security:
  networkPolicy:
    enabled: true
  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 1000

# values-production-rbc.yaml (RBC 고객용 오버라이드)
replicaCount: 3

image:
  repository: harbor.rbc.internal/cohere/north-platform
  tag: '3.5.0'
  pullPolicy: Always
  pullSecret: rbc-harbor-secret

modelServer:
  replicas: 2
  gpu:
    enabled: true
    count: 4
    type: nvidia-a100-80gb
  resources:
    requests:
      memory: '64Gi'
      cpu: '16'
    limits:
      memory: '128Gi'
      cpu: '32'

ingress:
  enabled: true
  className: nginx
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
    nginx.ingress.kubernetes.io/client-max-body-size: '100m'
  hosts:
    - host: north-ai.rbc.internal
  tls:
    enabled: true
    secretName: rbc-tls-secret

persistence:
  enabled: true
  storageClass: rbc-premium-ssd
  size: 500Gi

security:
  networkPolicy:
    enabled: true
  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 1000

# pre-install hook: 데이터베이스 마이그레이션
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    'helm.sh/hook': pre-install,pre-upgrade
    'helm.sh/hook-weight': '-5'
    'helm.sh/hook-delete-policy': hook-succeeded
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migration
          image: cohere/north-migration:latest
          command: ['./migrate', '--direction', 'up']

# 린트 검사
helm lint ./north-ai-platform

# 템플릿 렌더링 확인
helm template my-release ./north-ai-platform -f values-production-rbc.yaml

# 드라이런으로 설치 시뮬레이션
helm install my-release ./north-ai-platform --dry-run --debug

# 차트 테스트 실행
helm test my-release

# 차트 패키징
helm package ./north-ai-platform

# OCI Registry에 푸시
helm push north-ai-platform-2.1.0.tgz oci://registry.example.com/charts

# OCI Registry에서 설치
helm install north oci://registry.example.com/charts/north-ai-platform --version 2.1.0

# helmfile.yaml
repositories:
  - name: bitnami
    url: https://charts.bitnami.com/bitnami

releases:
  - name: north-platform
    chart: ./charts/north-ai-platform
    namespace: north-system
    values:
      - values/common.yaml
      - values/production.yaml

  - name: monitoring
    chart: prometheus-community/kube-prometheus-stack
    namespace: monitoring
    values:
      - values/monitoring.yaml

  - name: ingress
    chart: ingress-nginx/ingress-nginx
    namespace: ingress
    values:
      - values/ingress.yaml

┌─────────────────────────────────────────────┐
│                    VPC                       │
│  ┌──────────────────┐ ┌──────────────────┐  │
│  │  Public Subnet    │ │  Public Subnet    │  │
│  │  (AZ-1)          │ │  (AZ-2)          │  │
│  │  Load Balancer   │ │  Load Balancer   │  │
│  └────────┬─────────┘ └────────┬─────────┘  │
│           │                     │            │
│  ┌────────┴─────────┐ ┌────────┴─────────┐  │
│  │  Private Subnet   │ │  Private Subnet   │  │
│  │  (AZ-1)          │ │  (AZ-2)          │  │
│  │  K8s Worker      │ │  K8s Worker      │  │
│  │  Nodes           │ │  Nodes           │  │
│  └────────┬─────────┘ └────────┬─────────┘  │
│           │                     │            │
│  ┌────────┴─────────┐ ┌────────┴─────────┐  │
│  │  Data Subnet      │ │  Data Subnet      │  │
│  │  (AZ-1)          │ │  (AZ-2)          │  │
│  │  DB, Storage     │ │  DB, Storage     │  │
│  └──────────────────┘ └──────────────────┘  │
│                                              │
│  ┌──────────────────────────────────────┐   │
│  │  Private Endpoint (Storage/DB)       │   │
│  └──────────────────────────────────────┘   │
└─────────────────────────────────────────────┘

Azure: Managed Identity -> Pod Identity (AAD Pod Identity / Workload Identity)
AWS:   IAM Role -> IRSA (IAM Roles for Service Accounts)
GCP:   Service Account -> Workload Identity Federation

Developer -> Git Push -> GitHub/GitLab
                             │
                    ┌────────┴────────┐
                    │                 │
              CI Pipeline        ArgoCD/Flux
              (Build/Test)     (watches repo)
                    │                 │
              Container        Sync to K8s
              Registry         Cluster
                    │                 │
                    └────────┬────────┘
                             │
                      K8s Cluster
                    (Desired State)

name: North Platform CI/CD
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Helm Lint
        run: helm lint ./charts/north-ai-platform
      - name: Template Validation
        run: |
          helm template test ./charts/north-ai-platform \
            -f values/test.yaml \
            | kubectl apply --dry-run=client -f -

  security-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Trivy Chart Scan
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: config
          scan-ref: ./charts/north-ai-platform

  build-and-push:
    needs: [lint-and-test, security-scan]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build and Push Chart
        run: |
          helm package ./charts/north-ai-platform
          helm push north-ai-platform-*.tgz oci://registry.example.com/charts

# Azure AKS 클러스터 프로비저닝
resource "azurerm_kubernetes_cluster" "north" {
  name                = "north-aks-cluster"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  dns_prefix          = "north"
  kubernetes_version  = "1.29"

  default_node_pool {
    name       = "system"
    node_count = 3
    vm_size    = "Standard_D4s_v3"
  }

  identity {
    type = "SystemAssigned"
  }

  network_profile {
    network_plugin = "azure"
    network_policy = "calico"
  }

  private_cluster_enabled = true
}

# GPU 노드 풀
resource "azurerm_kubernetes_cluster_node_pool" "gpu" {
  name                  = "gpupool"
  kubernetes_cluster_id = azurerm_kubernetes_cluster.north.id
  vm_size              = "Standard_NC24ads_A100_v4"
  node_count           = 2

  node_taints = [
    "nvidia.com/gpu=present:NoSchedule"
  ]

  node_labels = {
    "accelerator" = "nvidia-a100"
  }
}

# HashiCorp Vault와 K8s 연동 (CSI Driver)
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: north-secrets
spec:
  provider: vault
  parameters:
    vaultAddress: 'https://vault.internal:8200'
    roleName: 'north-platform'
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/north/database"
        secretKey: "password"
      - objectName: "api-key"
        secretPath: "secret/data/north/api"
        secretKey: "key"

# Harbor 설치 (에어갭 환경용)
# 1. 인터넷 연결된 환경에서 모든 이미지 다운로드
docker pull goharbor/harbor-core:v2.10.0
docker pull goharbor/harbor-db:v2.10.0
docker pull goharbor/harbor-jobservice:v2.10.0
docker pull goharbor/harbor-portal:v2.10.0
docker pull goharbor/nginx-photon:v2.10.0
docker pull goharbor/registry-photon:v2.10.0

# 2. 이미지를 tar로 저장
docker save -o harbor-images.tar \
  goharbor/harbor-core:v2.10.0 \
  goharbor/harbor-db:v2.10.0 \
  goharbor/harbor-jobservice:v2.10.0 \
  goharbor/harbor-portal:v2.10.0

# 3. 에어갭 환경으로 물리적 미디어를 통해 전송

# 4. 에어갭 환경에서 이미지 로드
docker load -i harbor-images.tar

# 인터넷 환경에서 차트와 의존성 다운로드
helm pull oci://registry.example.com/charts/north-ai-platform --version 2.1.0
helm pull bitnami/postgresql --version 12.5.0
helm pull bitnami/redis --version 17.3.0

# 모든 컨테이너 이미지 목록 추출
helm template north ./north-ai-platform-2.1.0.tgz | \
  grep "image:" | awk '{print $2}' | sort -u > image-list.txt

# 이미지 일괄 다운로드 및 저장
while read -r image; do
  docker pull "$image"
done < image-list.txt

docker save -o north-platform-images.tar $(cat image-list.txt | tr '\n' ' ')

# 에어갭 환경에서 로드 및 태깅
docker load -i north-platform-images.tar
# Harbor 내부 레지스트리로 리태깅 후 푸시

# NVIDIA GPU Operator 설치 (에어갭 환경)
apiVersion: v1
kind: Namespace
metadata:
  name: gpu-operator
---
# NVIDIA Device Plugin DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin
  template:
    metadata:
      labels:
        name: nvidia-device-plugin
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: nvidia-device-plugin
          image: harbor.internal/nvidia/k8s-device-plugin:v0.14.0
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ['ALL']
          volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins

A100 80GB GPU
├── MIG 1g.10gb (소형 모델 서빙)
├── MIG 2g.20gb (중형 모델 서빙)
└── MIG 4g.40gb (대형 모델 서빙)

사용자 질의
    │
    ▼
┌──────────┐    ┌──────────────┐
│ Embed v3 │───▶│ Vector DB    │
│ (쿼리    │    │ (검색)       │
│  임베딩)  │    │ Pinecone/    │
└──────────┘    │ Weaviate/    │
                │ pgvector     │
                └──────┬───────┘
                       │ Top-K 문서
                       ▼
                ┌──────────────┐
                │ Rerank v3    │
                │ (관련성 재정렬)│
                └──────┬───────┘
                       │ 정제된 문서
                       ▼
                ┌──────────────┐
                │ Command R+   │
                │ (답변 생성)   │
                └──────────────┘
                       │
                       ▼
                   최종 답변

핵심 AI 서빙 메트릭:
├── 레이턴시
│   ├── Time to First Token (TTFT)
│   ├── Inter-Token Latency (ITL)
│   └── End-to-End Latency
├── 처리량
│   ├── Tokens per Second (TPS)
│   ├── Requests per Second (RPS)
│   └── Concurrent Requests
├── 리소스
│   ├── GPU Memory Utilization
│   ├── GPU Compute Utilization
│   └── KV Cache Hit Rate
└── 품질
    ├── Error Rate
    ├── Timeout Rate
    └── Queue Depth

1. 현재 상태 파악 (As-Is)
   - 기존 인프라 구성
   - 사용 중인 AI/ML 도구
   - 팀 구성과 역량

2. 목표 상태 정의 (To-Be)
   - 원하는 AI 활용 사례
   - 성능/확장성 요구사항
   - 보안/규제 요구사항

3. 갭 분석 (Gap Analysis)
   - 기술적 갭
   - 프로세스 갭
   - 인력 갭

4. 구현 계획 (Implementation Plan)
   - 단계별 마일스톤
   - 리스크와 완화 방안
   - 타임라인과 리소스

[장애 인지 시] (5분 이내)
"현재 [시스템명]에서 [증상]이 확인되었습니다.
원인 분석을 시작했으며, [예상 시간] 내에 업데이트를 드리겠습니다."

[원인 파악 시]
"원인이 [root cause]로 확인되었습니다.
[해결 방안]을 진행 중이며, 예상 복구 시간은 [ETA]입니다."

[해결 완료 시]
"[시간]에 서비스가 정상 복구되었습니다.
근본 원인: [상세 설명]
재발 방지: [대책]
상세 RCA(Root Cause Analysis) 보고서는 [기한]까지 전달하겠습니다."

llm-serving-platform/
  charts/
    llm-server/        # 메인 Helm Chart
    monitoring/         # 모니터링 Chart
  terraform/           # 인프라 프로비저닝
  docs/
    architecture.md
    deployment-guide.md
  scripts/
    setup.sh
    test.sh

1. 요약 (3줄)
   - 핵심 경력 연수 + 전문 분야
   - 가장 인상적인 성과 1개
   - FDE 역할에 대한 열정

2. 기술 스택
   - 프로그래밍: Python, Go, Bash
   - 인프라: K8s, Helm, Docker, Terraform
   - 클라우드: Azure, AWS, GCP
   - AI/ML: LLM 배포, vLLM, RAG
   - 도구: Git, ArgoCD, Prometheus, Grafana

3. 경력 사항 (STAR 형식, 3-5개)

4. 프로젝트 (GitHub 링크 포함, 2-3개)

5. 자격증
   - CKA, CKAD, 클라우드 자격증

6. 교육