Cohere Forward Deployed Engineer Complete Guide: Roadmap to Becoming an AI Platform Deployment Specialist

• 1. Understanding Cohere and the Agentic Platform Team
  • What Is Cohere
  • What Is the North AI Platform
  • The Agentic Platform Team's Mission
  • Key Client Analysis
  • Origins of the Forward Deployed Engineer Role
  • FDE vs SE vs SA Role Comparison
• 2. Line-by-Line JD Analysis
  • Key Responsibilities Breakdown
  • Required Qualifications Analysis
• 3. Tech Stack Deep Dive
  • 3-1. Kubernetes Deep Dive (Production Cluster Operations)
    • Cluster Architecture Understanding
    • Production Deployment Strategies
    • RBAC (Role-Based Access Control)
    • Network Isolation with NetworkPolicy
    • Resource Management
    • GPU Node Management
    • Monitoring Stack
    • Incident Response Procedures
  • 3-2. Helm Master Class
    • Helm Chart Structure
    • values.yaml Environment-Specific Override Strategy
    • Helm Hooks
    • Chart Testing
    • Helmfile for Multi-Chart Management
  • 3-3. Cloud Infrastructure (Azure/AWS/GCP)
    • Managed Kubernetes Comparison
    • Networking Core Concepts
    • IAM Strategy
  • 3-4. DevOps and CI/CD
    • GitOps Workflow
    • CI/CD Pipeline Example (GitHub Actions)
    • IaC: Kubernetes Cluster Provisioning with Terraform
    • Secret Management
  • 3-5. Private Cloud and On-Premises Deployment
    • Air-Gapped Environment Specifics
    • Harbor Mirror Registry Setup
    • Helm Chart Offline Bundle
    • On-Premises Kubernetes Options
    • Data Residency and Regulatory Compliance
    • GPU Infrastructure Management
  • 3-6. AI Model Deployment Technologies
    • LLM Serving Frameworks
    • RAG Architecture
    • Model Optimization Techniques
    • Monitoring Metrics
  • 3-7. Customer-Facing Skills (Soft Skills)
    • Technical Presentation Skills
    • Customer Requirements Gathering Framework
    • Language Skills
    • Incident Communication Template
• 4. 25 Expected Interview Questions
  • Kubernetes and Infrastructure (8 Questions)
  • Helm and Deployment Strategies (5 Questions)
  • Cloud and Networking (5 Questions)
  • AI Deployment and Troubleshooting (4 Questions)
  • Customer-Facing and Situational (3 Questions)
• 5. Eight-Month Study Roadmap
  • Month-by-Month Detailed Plan
• 6. Three Portfolio Project Ideas
  • Project 1: LLM Serving Pipeline with K8s + Helm
  • Project 2: Air-Gapped Environment Simulation Deployment
  • Project 3: Multi-Cloud AI Deployment Automation
• 7. Resume Writing Strategy
  • Organizing Experience in STAR Format
  • Essential Keywords for FDE Position
  • Emphasizing Customer-Facing Experience
  • Recommended Resume Structure
• 8. What It Is Like to Work at Cohere
  • Benefits and Culture
  • Growth Opportunities
  • Challenges
• 9. Quiz
• 10. References
  • Official Documentation
  • Certification Preparation
  • Learning Resources
  • Industry Trends
  • Communities
• Conclusion

1. Understanding Cohere and the Agentic Platform Team

What Is Cohere

Cohere is an enterprise AI company founded in Toronto in 2019. It was co-founded by Aidan Gomez (co-author of the Transformer paper "Attention Is All You Need" from Google Brain), Ivan Zhang, and Nick Frosst. While OpenAI targets the consumer market, Cohere has focused on enterprise B2B from day one, which is its key differentiator.

Key Product Lineup:

• Command R+: Enterprise-grade large language model (LLM) optimized for RAG (Retrieval-Augmented Generation)
• Embed v3: Multilingual embedding model supporting 100+ languages, specialized for search and classification
• Rerank v3: A reranking model that re-evaluates the relevance of search results
• Aya: An open-source multilingual model project supporting 101 languages

The reason Cohere holds a strong position in the enterprise market is its rigorous approach to data privacy. Its core strategy ensures customer data never leaves their environment through private cloud and on-premises deployments.

What Is the North AI Platform

The North AI Platform is Cohere's enterprise AI deployment platform. It evolved from what was previously known as the Cohere Toolkit, enabling companies to securely run Cohere's AI models within their own infrastructure.

Key Features of North:

• Complete AI stack deployment in private cloud and on-premises environments
• Kubernetes-based architecture deployable across diverse infrastructure
• Standardized deployment process through Helm Charts
• GPU resource management and model serving optimization
• Enterprise-grade security, monitoring, and logging integration

The Agentic Platform Team's Mission

The Agentic Platform team is one of the most customer-facing teams within Cohere. Their mission is to enable enterprise customers to operate AI agents safely and efficiently within their own environments.

AI agents go beyond simple chatbots — they use tools, perform multi-step reasoning, and automate real business workflows. Representative use cases include document analysis agents in financial institutions, medical record summarization agents in healthcare, and customer service automation agents in telecommunications.

Key Client Analysis

Cohere's enterprise clients span various industries, each with unique infrastructure requirements.

RBC (Royal Bank of Canada) - Finance

• Canada's largest bank, requiring compliance with global financial regulations (OSFI, GDPR, SOX)
• AI deployment in air-gapped environments is a core challenge
• Data residency requirements: mandatory data storage within Canadian territory
• Extreme security demands due to financial data sensitivity

Dell Technologies - IT/Hardware

• AI deployment on Dell's own server and storage infrastructure
• On-premises AI infrastructure combining Dell PowerEdge with NVIDIA GPUs
• Isolated AI workload management in multi-tenancy environments
• Secondary deployment scenarios providing AI solutions to Dell's own customers

LG CNS - IT Services/South Korea

• Compliance with South Korea's Personal Information Protection Act (PIPA)
• Korean language-specific AI model performance optimization requirements
• Serving diverse industry verticals: finance (LG affiliates), manufacturing, logistics
• Accommodating South Korea's strong preference for on-premises deployments

Origins of the Forward Deployed Engineer Role

The Forward Deployed Engineer (FDE) title was first created by Palantir Technologies. Inspired by the military term "forward deployed," it signifies placing engineers at the frontlines of customer engagements.

Unlike traditional software engineers who build products internally, FDEs directly understand customer environments and build solutions on top of them. Palantir's FDEs worked directly with the U.S. government, military, and intelligence agencies to deploy data analytics platforms.

As this model proved successful, many AI and data companies like Databricks, Scale AI, and Anyscale adopted similar roles, and Cohere followed suit.

FDE vs SE vs SA Role Comparison

2. Line-by-Line JD Analysis

Key Responsibilities Breakdown

"Lead North AI platform deployments across private cloud and on-premises environments"

This single line captures the essence of the position. "Lead" means not just participating but driving the entire deployment process. You must handle both private cloud (Azure Private Cloud, AWS Outposts, GCP Anthos, etc.) and on-premises (customer data centers).

In practice, this means:

• Customer infrastructure pre-assessment
• Deployment architecture design and documentation
• Kubernetes cluster setup and validation
• North platform deployment via Helm Charts
• GPU node configuration and model loading
• Integration testing and performance validation
• Handoff to customer operations team

"Partner with enterprise IT teams on infrastructure and security assessments"

Collaborating with enterprise IT teams demands communication skills as much as technical expertise. Large enterprise IT teams have strict security policies, network architectures, and change management processes.

Infrastructure assessment checklist:

• Network topology: VPC/VLAN configuration, subnets, firewall rules
• Security policies: authentication/authorization mechanisms, TLS certificate management
• Compute resources: CPU/GPU specifications, memory, storage IOPS
• Kubernetes environment: version, CNI plugin, ingress controller
• Regulatory compliance: data residency, audit logging, access control

"Design tailored deployment strategies ensuring data privacy compliance"

Customized deployment strategies vary per client. Financial institutions require air-gapped environments, healthcare needs HIPAA compliance, and EU customers must meet GDPR requirements.

Deployment strategy design considerations:

• Data flow: movement paths for training and inference data
• Encryption: at rest and in transit encryption
• Access control: who can access models and invoke APIs
• Audit trail: logging all access and changes
• Data retention: storage duration and deletion policies

"Troubleshoot deployment issues and minimize system downtime"

Troubleshooting ability in production environments is a core FDE competency. When failures occur in customer production environments, immediate response is required.

Common troubleshooting scenarios:

• Pod CrashLoopBackOff: OOM, configuration errors, dependency issues
• GPU allocation failures: driver mismatches, resource exhaustion
• Network connectivity issues: DNS, ingress, service mesh configuration
• Model loading failures: storage access permissions, model file corruption
• Performance degradation: resource contention, scaling issues

Required Qualifications Analysis

"Direct customer-facing experience"

Customer-facing experience is not just about talking to customers. You need to explain technically complex topics to non-technical decision-makers while engaging in deep technical discussions with engineering teams.

"Production Kubernetes cluster administration and Helm expertise"

This requires production-level K8s operational experience. Not personal project minikube, but managing multi-node clusters handling real traffic. Helm expertise means chart development and customization, not just basic usage.

"Cloud infrastructure (Azure, AWS, GCP), networking, virtualization"

Multi-cloud knowledge is essential. Since each customer uses different clouds, basic understanding of all three is needed. Networking (VPC, subnets, peering, private endpoints) and virtualization (VMware, KVM) knowledge are particularly important.

3. Tech Stack Deep Dive

3-1. Kubernetes Deep Dive (Production Cluster Operations)

Kubernetes is the most critical technology for this position. Production cluster management capability will make or break your candidacy.

Cluster Architecture Understanding

                     ┌─────────────────────────────┐
                     │       Control Plane          │
                     │  ┌─────────┐ ┌───────────┐  │
                     │  │kube-api │ │ scheduler │  │
                     │  │ server  │ │           │  │
                     │  └────┬────┘ └───────────┘  │
                     │  ┌────┴────┐ ┌───────────┐  │
                     │  │  etcd   │ │controller │  │
                     │  │         │ │ manager   │  │
                     │  └─────────┘ └───────────┘  │
                     └──────────────┬──────────────┘
                                    │
              ┌─────────────────────┼─────────────────────┐
              │                     │                     │
     ┌────────┴────────┐  ┌────────┴────────┐  ┌────────┴────────┐
     │   Worker Node 1  │  │   Worker Node 2  │  │  GPU Node (AI)  │
     │  ┌─────┐┌─────┐ │  │  ┌─────┐┌─────┐ │  │  ┌─────┐┌─────┐│
     │  │ Pod ││ Pod │ │  │  │ Pod ││ Pod │ │  │  │ Pod ││ Pod ││
     │  └─────┘└─────┘ │  │  └─────┘└─────┘ │  │  │ GPU ││ GPU ││
     │  kubelet+kproxy  │  │  kubelet+kproxy  │  │  └─────┘└─────┘│
     └──────────────────┘  └──────────────────┘  └─────────────────┘

Control Plane Core Components:

• kube-apiserver: Entry point for all API requests. Manages cluster state via REST API
• etcd: Distributed key-value store holding all cluster state. Backups are critical
• kube-scheduler: Schedules Pods onto appropriate nodes. Places GPU-requesting Pods on GPU nodes
• kube-controller-manager: Manages state of Deployments, ReplicaSets, DaemonSets, etc.

Production Deployment Strategies

# Rolling Update - Most common
apiVersion: apps/v1
kind: Deployment
metadata:
  name: north-ai-api
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    spec:
      containers:
        - name: north-api
          image: cohere/north-api:v2.1.0
          resources:
            requests:
              memory: '4Gi'
              cpu: '2'
            limits:
              memory: '8Gi'
              cpu: '4'
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 30

Deployment Strategy Comparison:

RBAC (Role-Based Access Control)

RBAC configuration is fundamental to security in enterprise environments.

# Per-customer namespace isolation
apiVersion: v1
kind: Namespace
metadata:
  name: customer-rbc
  labels:
    customer: rbc
    environment: production
---
# Role definition following least privilege principle
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: customer-rbc
  name: north-deployer
rules:
  - apiGroups: ['apps']
    resources: ['deployments', 'statefulsets']
    verbs: ['get', 'list', 'watch', 'create', 'update', 'patch']
  - apiGroups: ['']
    resources: ['pods', 'services', 'configmaps', 'secrets']
    verbs: ['get', 'list', 'watch', 'create', 'update']
  - apiGroups: ['']
    resources: ['pods/log']
    verbs: ['get']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: customer-rbc
  name: north-deployer-binding
subjects:
  - kind: ServiceAccount
    name: north-deploy-sa
    namespace: customer-rbc
roleRef:
  kind: Role
  name: north-deployer
  apiGroup: rbac.authorization.k8s.io

Network Isolation with NetworkPolicy

# Allow only inter-North AI platform Pod communication
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: north-platform-policy
  namespace: customer-rbc
spec:
  podSelector:
    matchLabels:
      app: north-platform
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: north-platform
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: north-platform
      ports:
        - protocol: TCP
          port: 8080
    - to: # Allow DNS
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53

Resource Management

# LimitRange for namespace defaults
apiVersion: v1
kind: LimitRange
metadata:
  name: north-limits
  namespace: customer-rbc
spec:
  limits:
    - default:
        memory: '2Gi'
        cpu: '1'
      defaultRequest:
        memory: '512Mi'
        cpu: '250m'
      type: Container
---
# ResourceQuota for namespace-wide limits
apiVersion: v1
kind: ResourceQuota
metadata:
  name: north-quota
  namespace: customer-rbc
spec:
  hard:
    requests.cpu: '32'
    requests.memory: '64Gi'
    limits.cpu: '64'
    limits.memory: '128Gi'
    requests.nvidia.com/gpu: '8'
    pods: '50'

GPU Node Management

GPU management is especially critical for AI model serving.

# Tolerations + nodeSelector for GPU node placement
apiVersion: apps/v1
kind: Deployment
metadata:
  name: north-model-server
spec:
  template:
    spec:
      nodeSelector:
        accelerator: nvidia-a100
      tolerations:
        - key: 'nvidia.com/gpu'
          operator: 'Exists'
          effect: 'NoSchedule'
      containers:
        - name: model-server
          image: cohere/north-model:latest
          resources:
            limits:
              nvidia.com/gpu: 4
          volumeMounts:
            - name: model-storage
              mountPath: /models
            - name: shm
              mountPath: /dev/shm
      volumes:
        - name: model-storage
          persistentVolumeClaim:
            claimName: model-pvc
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: '16Gi'

Monitoring Stack

# Prometheus ServiceMonitor configuration
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: north-platform-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: north-platform
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

Key monitoring metrics:

• Pod level: CPU/memory utilization, restart count, OOM kill count
• Node level: Node availability, disk usage, GPU utilization
• Cluster level: Scheduling latency, etcd latency, API server response time
• AI workload: Inference latency, tokens/sec throughput, queue depth

Incident Response Procedures

# 1. Identify problematic node
kubectl get nodes -o wide
kubectl describe node problematic-node

# 2. Cordon node (prevent new Pod scheduling)
kubectl cordon problematic-node

# 3. Safely move existing workloads (drain)
kubectl drain problematic-node --ignore-daemonsets --delete-emptydir-data

# 4. Restore node after fixing the issue
kubectl uncordon problematic-node

# 5. Verify Pod status
kubectl get pods -n customer-rbc -o wide
kubectl logs -n customer-rbc pod-name --previous

Recommended Certification Path:

1. CKA (Certified Kubernetes Administrator): Cluster management focused - Essential
2. CKAD (Certified Kubernetes Application Developer): Application deployment - Recommended
3. CKS (Certified Kubernetes Security Specialist): Security - Preferred

3-2. Helm Master Class

Helm is Kubernetes' package manager and a tool you will use daily in this position. Since the North AI platform's deployment unit is a Helm Chart, chart development capability is essential.

Helm Chart Structure

north-ai-platform/
  Chart.yaml          # Chart metadata
  Chart.lock          # Dependency lock file
  values.yaml         # Default values
  values-production.yaml   # Production override
  values-staging.yaml      # Staging override
  templates/
    _helpers.tpl      # Common template functions
    deployment.yaml   # Deployment resource
    service.yaml      # Service resource
    ingress.yaml      # Ingress resource
    configmap.yaml    # ConfigMap
    secret.yaml       # Secret
    hpa.yaml          # HorizontalPodAutoscaler
    pdb.yaml          # PodDisruptionBudget
    networkpolicy.yaml
    serviceaccount.yaml
    NOTES.txt         # Post-install instructions
  charts/             # Sub-charts (dependencies)
  tests/
    test-connection.yaml

values.yaml Environment-Specific Override Strategy

# values.yaml (defaults)
replicaCount: 1
image:
  repository: cohere/north-platform
  tag: 'latest'
  pullPolicy: IfNotPresent

modelServer:
  replicas: 1
  gpu:
    enabled: false
    count: 1
    type: ''
  resources:
    requests:
      memory: '8Gi'
      cpu: '4'
    limits:
      memory: '16Gi'
      cpu: '8'

ingress:
  enabled: true
  className: nginx
  tls:
    enabled: true

monitoring:
  enabled: true
  prometheus:
    scrape: true

security:
  networkPolicy:
    enabled: true
  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 1000

# values-production-rbc.yaml (RBC customer override)
replicaCount: 3

image:
  repository: harbor.rbc.internal/cohere/north-platform
  tag: '3.5.0'
  pullPolicy: Always
  pullSecret: rbc-harbor-secret

modelServer:
  replicas: 2
  gpu:
    enabled: true
    count: 4
    type: nvidia-a100-80gb
  resources:
    requests:
      memory: '64Gi'
      cpu: '16'
    limits:
      memory: '128Gi'
      cpu: '32'

ingress:
  enabled: true
  className: nginx
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
    nginx.ingress.kubernetes.io/client-max-body-size: '100m'
  hosts:
    - host: north-ai.rbc.internal
  tls:
    enabled: true
    secretName: rbc-tls-secret

persistence:
  enabled: true
  storageClass: rbc-premium-ssd
  size: 500Gi

Helm Hooks

# pre-install hook: database migration
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
  annotations:
    'helm.sh/hook': pre-install,pre-upgrade
    'helm.sh/hook-weight': '-5'
    'helm.sh/hook-delete-policy': hook-succeeded
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migration
          image: cohere/north-migration:latest
          command: ['./migrate', '--direction', 'up']

Chart Testing

# Lint check
helm lint ./north-ai-platform

# Template rendering verification
helm template my-release ./north-ai-platform -f values-production-rbc.yaml

# Dry-run install simulation
helm install my-release ./north-ai-platform --dry-run --debug

# Run chart tests
helm test my-release

Helmfile for Multi-Chart Management

# helmfile.yaml
repositories:
  - name: bitnami
    url: https://charts.bitnami.com/bitnami

releases:
  - name: north-platform
    chart: ./charts/north-ai-platform
    namespace: north-system
    values:
      - values/common.yaml
      - values/production.yaml

  - name: monitoring
    chart: prometheus-community/kube-prometheus-stack
    namespace: monitoring
    values:
      - values/monitoring.yaml

  - name: ingress
    chart: ingress-nginx/ingress-nginx
    namespace: ingress
    values:
      - values/ingress.yaml

3-3. Cloud Infrastructure (Azure/AWS/GCP)

Managed Kubernetes Comparison

Networking Core Concepts

┌─────────────────────────────────────────────┐
│                    VPC                       │
│  ┌──────────────────┐ ┌──────────────────┐  │
│  │  Public Subnet    │ │  Public Subnet    │  │
│  │  (AZ-1)          │ │  (AZ-2)          │  │
│  │  Load Balancer   │ │  Load Balancer   │  │
│  └────────┬─────────┘ └────────┬─────────┘  │
│           │                     │            │
│  ┌────────┴─────────┐ ┌────────┴─────────┐  │
│  │  Private Subnet   │ │  Private Subnet   │  │
│  │  (AZ-1)          │ │  (AZ-2)          │  │
│  │  K8s Worker      │ │  K8s Worker      │  │
│  │  Nodes           │ │  Nodes           │  │
│  └────────┬─────────┘ └────────┬─────────┘  │
│           │                     │            │
│  ┌────────┴─────────┐ ┌────────┴─────────┐  │
│  │  Data Subnet      │ │  Data Subnet      │  │
│  │  (AZ-1)          │ │  (AZ-2)          │  │
│  │  DB, Storage     │ │  DB, Storage     │  │
│  └──────────────────┘ └──────────────────┘  │
│                                              │
│  ┌──────────────────────────────────────┐   │
│  │  Private Endpoint (Storage/DB)       │   │
│  └──────────────────────────────────────┘   │
└─────────────────────────────────────────────┘

Key Networking Elements:

• VPC Peering / Transit Gateway: Connecting multiple VPCs
• Private Endpoint / PrivateLink: Service access without traversing public internet
• Network Security Group / Security Group: Inbound/outbound traffic control
• DNS: Internal service name resolution with Private DNS Zones

IAM Strategy

Workload identity mapping across clouds:

Azure: Managed Identity -> Pod Identity (AAD Pod Identity / Workload Identity)
AWS:   IAM Role -> IRSA (IAM Roles for Service Accounts)
GCP:   Service Account -> Workload Identity Federation

Recommended Certifications:

• Azure: AZ-104 (Azure Administrator) or AZ-305 (Solutions Architect)
• AWS: SAA-C03 (Solutions Architect Associate)
• GCP: Professional Cloud Architect

3-4. DevOps and CI/CD

GitOps Workflow

Developer -> Git Push -> GitHub/GitLab
                             │
                    ┌────────┴────────┐
                    │                 │
              CI Pipeline        ArgoCD/Flux
              (Build/Test)     (watches repo)
                    │                 │
              Container        Sync to K8s
              Registry         Cluster
                    │                 │
                    └────────┬────────┘
                             │
                      K8s Cluster
                    (Desired State)

CI/CD Pipeline Example (GitHub Actions)

name: North Platform CI/CD
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Helm Lint
        run: helm lint ./charts/north-ai-platform
      - name: Template Validation
        run: |
          helm template test ./charts/north-ai-platform \
            -f values/test.yaml \
            | kubectl apply --dry-run=client -f -

  security-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Trivy Chart Scan
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: config
          scan-ref: ./charts/north-ai-platform

  build-and-push:
    needs: [lint-and-test, security-scan]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build and Push Chart
        run: |
          helm package ./charts/north-ai-platform
          helm push north-ai-platform-*.tgz oci://registry.example.com/charts

IaC: Kubernetes Cluster Provisioning with Terraform

# Azure AKS cluster provisioning
resource "azurerm_kubernetes_cluster" "north" {
  name                = "north-aks-cluster"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  dns_prefix          = "north"
  kubernetes_version  = "1.29"

  default_node_pool {
    name       = "system"
    node_count = 3
    vm_size    = "Standard_D4s_v3"
  }

  identity {
    type = "SystemAssigned"
  }

  network_profile {
    network_plugin = "azure"
    network_policy = "calico"
  }

  private_cluster_enabled = true
}

# GPU node pool
resource "azurerm_kubernetes_cluster_node_pool" "gpu" {
  name                  = "gpupool"
  kubernetes_cluster_id = azurerm_kubernetes_cluster.north.id
  vm_size              = "Standard_NC24ads_A100_v4"
  node_count           = 2

  node_taints = [
    "nvidia.com/gpu=present:NoSchedule"
  ]

  node_labels = {
    "accelerator" = "nvidia-a100"
  }
}

Secret Management

# HashiCorp Vault with K8s integration (CSI Driver)
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: north-secrets
spec:
  provider: vault
  parameters:
    vaultAddress: 'https://vault.internal:8200'
    roleName: 'north-platform'
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/north/database"
        secretKey: "password"
      - objectName: "api-key"
        secretPath: "secret/data/north/api"
        secretKey: "key"

3-5. Private Cloud and On-Premises Deployment

This section covers the most critical differentiating skill for this position. Many engineers can operate K8s on public cloud, but air-gapped deployment experience is rare.

Air-Gapped Environment Specifics

An air-gapped environment is a network completely isolated from the external internet. It is primarily used in finance (RBC), military, and healthcare organizations.

Key Challenges in Air-Gapped Environments:

1. Container image delivery: No access to Docker Hub, GitHub Container Registry
2. Package installation: No access to apt, yum, pip, npm repositories
3. Helm Chart downloads: No access to chart repositories
4. Certificate management: Cannot use external CAs like Let's Encrypt
5. Time synchronization: NTP server access may be restricted

Harbor Mirror Registry Setup

# Harbor installation (for air-gapped environments)
# 1. Download all images on an internet-connected machine
docker pull goharbor/harbor-core:v2.10.0
docker pull goharbor/harbor-db:v2.10.0
docker pull goharbor/harbor-jobservice:v2.10.0
docker pull goharbor/harbor-portal:v2.10.0
docker pull goharbor/nginx-photon:v2.10.0
docker pull goharbor/registry-photon:v2.10.0

# 2. Save images as tar
docker save -o harbor-images.tar \
  goharbor/harbor-core:v2.10.0 \
  goharbor/harbor-db:v2.10.0 \
  goharbor/harbor-jobservice:v2.10.0 \
  goharbor/harbor-portal:v2.10.0

# 3. Transfer to air-gapped environment via physical media

# 4. Load images in air-gapped environment
docker load -i harbor-images.tar

Helm Chart Offline Bundle

# Download charts and dependencies on internet-connected machine
helm pull oci://registry.example.com/charts/north-ai-platform --version 2.1.0
helm pull bitnami/postgresql --version 12.5.0
helm pull bitnami/redis --version 17.3.0

# Extract all container image list
helm template north ./north-ai-platform-2.1.0.tgz | \
  grep "image:" | awk '{print $2}' | sort -u > image-list.txt

# Batch download and save images
while read -r image; do
  docker pull "$image"
done < image-list.txt

docker save -o north-platform-images.tar $(cat image-list.txt | tr '\n' ' ')

# Load and retag in air-gapped environment
docker load -i north-platform-images.tar
# Retag to internal Harbor registry and push

On-Premises Kubernetes Options

Data Residency and Regulatory Compliance

GPU Infrastructure Management

# NVIDIA GPU Operator installation (air-gapped)
apiVersion: v1
kind: Namespace
metadata:
  name: gpu-operator
---
# NVIDIA Device Plugin DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin
  template:
    metadata:
      labels:
        name: nvidia-device-plugin
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: nvidia-device-plugin
          image: harbor.internal/nvidia/k8s-device-plugin:v0.14.0
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ['ALL']
          volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins

NVIDIA MIG (Multi-Instance GPU) Configuration:

MIG allows partitioning large GPUs like the A100 into multiple isolated instances to serve multiple models simultaneously.

A100 80GB GPU
├── MIG 1g.10gb (small model serving)
├── MIG 2g.20gb (medium model serving)
└── MIG 4g.40gb (large model serving)

3-6. AI Model Deployment Technologies

LLM Serving Frameworks

RAG Architecture

User Query
    │
    ▼
┌──────────┐    ┌──────────────┐
│ Embed v3 │───▶│ Vector DB    │
│ (query   │    │ (search)     │
│ embedding)│    │ Pinecone/    │
└──────────┘    │ Weaviate/    │
                │ pgvector     │
                └──────┬───────┘
                       │ Top-K docs
                       ▼
                ┌──────────────┐
                │ Rerank v3    │
                │ (relevance   │
                │  reranking)  │
                └──────┬───────┘
                       │ Refined docs
                       ▼
                ┌──────────────┐
                │ Command R+   │
                │ (answer gen) │
                └──────────────┘
                       │
                       ▼
                  Final Answer

Model Optimization Techniques

• Quantization: FP32 -> FP16 -> INT8 -> INT4 to reduce model size and memory
• Knowledge Distillation: Transferring knowledge from a large model to a smaller one
• KV Cache Optimization: Memory efficiency improvement via PagedAttention
• Batch Processing: Throughput maximization via Continuous Batching

Monitoring Metrics

Core AI Serving Metrics:
├── Latency
│   ├── Time to First Token (TTFT)
│   ├── Inter-Token Latency (ITL)
│   └── End-to-End Latency
├── Throughput
│   ├── Tokens per Second (TPS)
│   ├── Requests per Second (RPS)
│   └── Concurrent Requests
├── Resources
│   ├── GPU Memory Utilization
│   ├── GPU Compute Utilization
│   └── KV Cache Hit Rate
└── Quality
    ├── Error Rate
    ├── Timeout Rate
    └── Queue Depth

3-7. Customer-Facing Skills (Soft Skills)

The FDE position values customer-facing ability as much as technical capability.

Technical Presentation Skills

• Audience adaptation: Architecture level for CTOs, implementation level for IT teams
• Demo preparation: Always have backup recordings for live demos
• Handling questions: Honestly admit unknowns and promise follow-up

Customer Requirements Gathering Framework

1. Current State Assessment (As-Is)
   - Existing infrastructure configuration
   - Current AI/ML tools in use
   - Team composition and capabilities

2. Target State Definition (To-Be)
   - Desired AI use cases
   - Performance/scalability requirements
   - Security/regulatory requirements

3. Gap Analysis
   - Technical gaps
   - Process gaps
   - Talent gaps

4. Implementation Plan
   - Phased milestones
   - Risks and mitigation strategies
   - Timeline and resources

Language Skills

As this position is Japan-based:

• English: Technical documentation, global team communication (required)
• Japanese: Japanese customer engagement (highly advantageous)
• Korean: LG CNS and Korean customer engagement (bonus)

Incident Communication Template

[Upon incident detection] (within 5 minutes)
"We have identified [symptom] in [system name].
Root cause analysis has begun, and we will provide an update within [estimated time]."

[When cause is identified]
"The root cause has been identified as [root cause].
[Resolution approach] is in progress, with an estimated recovery time of [ETA]."

[Upon resolution]
"Service was restored to normal at [time].
Root cause: [detailed explanation]
Prevention measures: [countermeasures]
A detailed RCA (Root Cause Analysis) report will be delivered by [deadline]."

4. 25 Expected Interview Questions

Kubernetes and Infrastructure (8 Questions)

Q1. Describe the incident response procedure when a node fails in a production Kubernetes cluster.

Key answer points: Failure detection (monitoring alert) -> Impact scope assessment (which Pods are on the node) -> Cordon to block new scheduling -> Drain to migrate workloads -> Fix or replace node -> Uncordon to restore -> Post-mortem analysis

Q2. Explain your etcd backup and recovery strategy.

Key answer points: Regular etcd snapshots, backup frequency criteria, recovery procedure, maintaining quorum in distributed environments, etcd performance impact on the entire cluster

Q3. Describe the debugging procedure when a Pod is stuck in Pending state.

Key answer points: Check events with kubectl describe pod -> Verify resource availability (CPU/memory/GPU) -> Check PVC binding -> Verify nodeSelector/affinity/taint -> Check scheduler logs

Q4. Explain how to control microservice communication using NetworkPolicy.

Key answer points: Set default deny-all policy -> Whitelist only required communication -> Namespace isolation -> Restrict external communication with egress control

Q5. Explain how to manage GPU resources in Kubernetes.

Key answer points: NVIDIA Device Plugin, resource limits, nodeSelector, tolerations, MIG configuration, monitoring (DCGM exporter)

Q6. Explain the role of PodDisruptionBudget (PDB) and configuration strategies.

Key answer points: Ensuring minimum Pod availability during voluntary disruptions (upgrades, scale-down), minAvailable vs maxUnavailable strategies, relationship with StatefulSets

Q7. Explain the differences between Horizontal Pod Autoscaler and Vertical Pod Autoscaler and their respective use cases.

Key answer points: HPA adjusts Pod count, VPA adjusts resource requests. AI inference services primarily use horizontal scaling for GPU Pods. Custom metrics-based scaling (queue depth, latency)

Q8. Explain how to implement namespace-level isolation in a multi-tenancy environment.

Key answer points: Namespace + RBAC + NetworkPolicy + ResourceQuota + LimitRange + Pod Security Standards combination

Helm and Deployment Strategies (5 Questions)

Q9. Explain the values.yaml override strategy for Helm Charts. How do you manage multiple environments (dev/staging/prod)?

Key answer points: Base values.yaml + environment-specific override files, Helmfile usage, secret management strategy (Sealed Secrets, SOPS)

Q10. Share your experience with deployment automation using Helm Hooks.

Key answer points: pre-install/pre-upgrade for DB migrations, post-install for initial setup, hook weight for execution order control

Q11. Describe the entire process of deploying a Helm Chart to an air-gapped environment.

Key answer points: Chart packaging -> Image list extraction -> Image download/save -> Physical media transfer -> Internal registry load -> Modify values (image path changes) -> Install

Q12. Explain how to implement Canary deployment with Kubernetes and Helm.

Key answer points: Deploy canary version as separate Deployment, use Service selector, control traffic ratios with Istio VirtualService, metrics-based auto-promotion/rollback

Q13. Describe the strategy for managing CRDs (Custom Resource Definitions) in Helm Charts.

Key answer points: Using crds/ directory, CRD upgrade considerations, Helm's policy of not deleting CRDs, pattern of managing CRDs as separate charts

Cloud and Networking (5 Questions)

Q14. Compare the pros and cons of each cloud provider's Managed Kubernetes service (Azure, AWS, GCP).

Key answer points: Content from section 3-3 comparison table + insights from actual operational experience

Q15. Explain the networking elements to consider when designing a VPC for AI workloads.

Key answer points: Subnet separation (public/private/data), bandwidth requirements (inter-GPU node communication), Private Endpoint, NAT Gateway, DNS resolution

Q16. Explain how to establish a secure connection between on-premises and cloud in a hybrid cloud environment.

Key answer points: VPN Gateway, ExpressRoute/Direct Connect/Interconnect, mTLS, certificate management, bandwidth planning

Q17. Share your experience managing multi-cloud infrastructure with Terraform.

Key answer points: Provider-specific module separation, remote state management (Backend), workspace strategy, module reuse patterns

Q18. Describe your experience setting up storage access via Private Endpoints.

Key answer points: Azure Private Endpoint / AWS VPC Endpoint / GCP Private Service Connect, DNS configuration, network rules

AI Deployment and Troubleshooting (4 Questions)

Q19. Explain the key challenges and solutions when serving large LLMs in a Kubernetes environment.

Key answer points: GPU memory management, model loading time, cold start issues, batch processing optimization, model update strategies (minimizing downtime)

Q20. Share your experience designing and deploying a RAG (Retrieval-Augmented Generation) system architecture.

Key answer points: Vector DB selection criteria, embedding model serving, search-reranking-generation pipeline, document chunking strategy, performance tuning

Q21. Describe the debugging process when OOM (Out of Memory) errors occur repeatedly during AI model serving.

Key answer points: Distinguish GPU memory vs system memory, check GPU usage with nvidia-smi, adjust batch size, apply quantization, limit KV cache size, shared memory configuration

Q22. Explain the strategy for performing zero-downtime model updates.

Key answer points: Prepare new model version with Blue-Green deployment, confirm model loading completion via Health Check before traffic switching, rollback plan, A/B testing possibility

Customer-Facing and Situational (3 Questions)

Q23. How would you respond to an urgent failure in a customer's production environment?

Key answer points: Immediate response (within 10 minutes), impact scope assessment, temporary fix (workaround) application, customer communication (regular updates), root cause analysis, RCA report, prevention measures

Q24. How would you approach deploying the North Platform when the customer's IT team has limited Kubernetes experience?

Key answer points: Customer team capability assessment, phased training plan development, documentation (deployment guide, operations guide), operations handoff plan, post-deployment support period

Q25. How would you respond when a customer presents technically impossible requirements (e.g., real-time model updates in an air-gapped environment)?

Key answer points: Identify the essential need behind the requirement, propose alternatives (periodic update cycles, semi-air-gapped configuration), clearly explain trade-offs, document technical rationale

5. Eight-Month Study Roadmap

Month-by-Month Detailed Plan

Month 1: Kubernetes Basics

• Week 1-2: Understand K8s architecture, master kubectl
• Week 3-4: Hands-on with Deployment, Service, ConfigMap, Secret
• Daily 1 hour CKA practice problems
• Tools: minikube, kind

Month 2: Kubernetes Advanced + CKA

• Week 1: RBAC, ServiceAccount
• Week 2: NetworkPolicy, Ingress
• Week 3: PV/PVC, StorageClass, StatefulSet
• Week 4: Take CKA exam
• Tools: kubeadm, Vagrant

Month 3: Helm + Cloud Introduction

• Week 1-2: Helm basics, analyze existing charts
• Week 3: Custom chart development, template engine
• Week 4: First experience with AKS or EKS
• Project: Package a microservices app as a Helm Chart

Month 4: Cloud Infrastructure + Terraform

• Week 1: VPC design, subnets, security groups
• Week 2: IAM, service accounts, workload identity
• Week 3-4: Terraform basics, module authoring
• Project: Build a private K8s cluster with Terraform

Month 5: CI/CD + GitOps

• Week 1: Build GitHub Actions pipeline
• Week 2: ArgoCD installation and configuration
• Week 3: Container image scanning, Trivy
• Week 4: Integrated CI/CD + GitOps pipeline
• Project: Push-to-Deploy automation implementation

Month 6: Air-Gapped/On-Premises

• Week 1: Harbor installation and operation
• Week 2: Create offline image/chart bundles
• Week 3: Build air-gapped K8s cluster with RKE2
• Week 4: Security hardening (Falco, OPA/Gatekeeper)
• Project: Deploy app in fully air-gapped environment

Month 7: AI Model Deployment

• Week 1: LLM serving with vLLM
• Week 2: GPU management, NVIDIA Device Plugin
• Week 3: Build RAG pipeline
• Week 4: Monitoring and optimization
• Project: Build LLM serving on K8s + Helm

Month 8: Integration + Interview Prep

• Week 1-2: Complete portfolio project
• Week 3: Technical interview mock practice
• Week 4: Behavioral interview prep, resume final review
• Daily: Practice the 25 interview questions

6. Three Portfolio Project Ideas

Project 1: LLM Serving Pipeline with K8s + Helm

Goal: Build a complete pipeline for deploying and operating an open-source LLM in a Kubernetes environment using Helm Charts

Tech Stack: Kubernetes, Helm, vLLM, Prometheus, Grafana, NVIDIA GPU Operator

Implementation:

• Develop custom Helm Chart for vLLM-based model serving
• GPU node management and auto-scaling configuration
• Build inference metrics dashboard with Prometheus + Grafana
• Health check and auto-recovery mechanisms
• Blue-Green deployment strategy for model updates
• Include architecture diagram and deployment guide in README

Project 2: Air-Gapped Environment Simulation Deployment

Goal: Simulate a completely internet-isolated environment and practice the entire process of deploying AI services within it

Tech Stack: Vagrant, RKE2, Harbor, Helm, container image bundling

Implementation:

• Build network-isolated VM environment with Vagrant
• Install and configure Harbor mirror registry
• Build air-gapped K8s cluster with RKE2
• Package all images and charts as offline bundles
• Automate deployment via bundle scripts
• Manage TLS certificates with internal CA

Project 3: Multi-Cloud AI Deployment Automation

Goal: Build an automation system that deploys the same AI service to Azure, AWS, and GCP using Terraform and Helm

Tech Stack: Terraform, Helm, Helmfile, GitHub Actions, AKS/EKS/GKE

Implementation:

• Provision cloud-specific K8s clusters with Terraform modules
• Cloud-agnostic application deployment with Helmfile
• CI/CD pipeline integration with GitHub Actions
• Abstract cloud differences (storage, IAM, networking)
• Cost comparison analysis documentation
• Include disaster recovery (DR) strategy

7. Resume Writing Strategy

Organizing Experience in STAR Format

Each experience on your resume should be structured using the STAR (Situation-Task-Action-Result) format.

Good Example:

"Led migration from legacy VM-based deployment to Kubernetes (S/T). Developed Helm Charts and built ArgoCD-based GitOps pipeline to automate deployments (A). Reduced deployment time by 87% from 2 hours to 15 minutes and decreased deployment-related incidents from 3 per month to zero (R)."

Essential Keywords for FDE Position

Keywords to include in your resume:

• Infrastructure: Kubernetes, Helm, Terraform, Docker, CI/CD
• Cloud: Azure/AWS/GCP, VPC, IAM, private cloud
• Security: RBAC, NetworkPolicy, TLS, air-gapped, regulatory compliance
• AI/ML: LLM deployment, GPU management, model serving, RAG
• Soft skills: Customer-facing, technical presentations, documentation, troubleshooting

Emphasizing Customer-Facing Experience

The most differentiating factor for an FDE position is customer-facing experience.

Points to highlight:

• Direct customer communication experience (technical meetings, demos, training)
• Direct deployment/operations at customer sites
• Incident response and RCA (Root Cause Analysis) experience
• Technical documentation creation and delivery
• Explaining technical concepts to non-technical decision-makers

Recommended Resume Structure

1. Summary (3 lines)
   - Core experience years + specialty
   - Most impressive achievement (1)
   - Passion for the FDE role

2. Technical Skills
   - Programming: Python, Go, Bash
   - Infrastructure: K8s, Helm, Docker, Terraform
   - Cloud: Azure, AWS, GCP
   - AI/ML: LLM deployment, vLLM, RAG
   - Tools: Git, ArgoCD, Prometheus, Grafana

3. Professional Experience (STAR format, 3-5 items)

4. Projects (with GitHub links, 2-3 items)

5. Certifications
   - CKA, CKAD, cloud certifications

6. Education

8. What It Is Like to Work at Cohere

Benefits and Culture

Cohere's benefits are among the best in the AI startup space.

Key Benefits:

• 6 weeks vacation: Double the statutory vacation in many countries
• 100% parental leave top-up (6 months): Full salary guaranteed on top of government support
• Remote work flexibility: Work from anywhere in Japan, travel to EMEA/APAC regions
• Health insurance: Comprehensive health, dental, and vision coverage
• Learning support: Certification, conference, and book purchase support
• Stock options: Wealth growth opportunity with startup growth

Growth Opportunities

Frontline AI Experience:

• Hands-on experience deploying world-class LLMs
• Learning the latest trends and technologies in enterprise AI deployment
• Exposure to AI applications across finance, healthcare, telecom, and more

Global Network:

• Collaboration with global enterprise clients like RBC, Dell, LG CNS
• Working with multinational teams (Canada, US, Japan, Europe)
• Opportunities to participate in global AI conferences and communities

Challenges

Being honest about the position's challenges is important.

Travel (20-40%)

• You may spend 1-2 weeks per month at customer sites
• Travel within Japan and across the Asia-Pacific region
• Balancing remote work with on-site requirements

Customer Site Pressure

• Production environment failures at customer sites create high stress
• Must quickly adapt to diverse environments across different customers
• Navigating between technical limitations and customer expectations

Breadth of Technical Scope

• Must handle K8s, Helm, cloud, networking, security, and AI
• Continuous learning is mandatory
• Maintaining both breadth and depth simultaneously is the challenge

9. Quiz

Let us review what we have learned.

10. References

Official Documentation

1. Cohere Documentation - docs.cohere.com - Cohere API and model guides
2. Kubernetes Documentation - kubernetes.io/docs - Complete Kubernetes reference
3. Helm Documentation - helm.sh/docs - Helm chart development guide
4. NVIDIA GPU Operator - docs.nvidia.com/datacenter/cloud-native - GPU management guide

Certification Preparation

5. CKA Exam Guide - training.linuxfoundation.org - CNCF Certified K8s Administrator
6. CKAD Exam Guide - training.linuxfoundation.org - CNCF Certified K8s Developer
7. Azure AZ-104 - learn.microsoft.com - Azure Administrator certification
8. AWS SAA-C03 - aws.amazon.com/certification - AWS Solutions Architect

Learning Resources

9. Kubernetes The Hard Way - Kelsey Hightower's deep K8s learning
10. Helm Chart Development Best Practices - helm.sh/docs/chart_best_practices
11. Terraform Up and Running - by Yevgeniy Brikman, the IaC bible
12. vLLM Documentation - docs.vllm.ai - LLM serving framework

Industry Trends

13. Cohere Blog - cohere.com/blog - Latest model and technology updates
14. CNCF Landscape - landscape.cncf.io - Cloud native ecosystem map
15. AI Infrastructure Alliance - ai-infrastructure.org - AI infrastructure trends

Communities

16. CNCF Slack - slack.cncf.io - Cloud native community
17. Kubernetes subreddit - reddit.com/r/kubernetes - K8s community
18. MLOps Community - mlops.community - ML operations community

Conclusion

The Cohere Forward Deployed Engineer (Infrastructure Specialist) position is one of the most exciting roles in the AI era. It is not just about writing code — it is about bringing world-class AI technology directly to enterprise environments.

To summarize what we covered in this guide:

• Cohere is a company focused on enterprise AI, with the North platform as its core product
• FDE is a unique role that deploys directly at customer sites
• Kubernetes and Helm are the technical core, and air-gapped deployment experience is a major differentiator
• Eight months of systematic study can equip you with the necessary skills
• Customer-facing ability is as important as technical skills

Following this roadmap with systematic preparation will provide a solid foundation for starting your career as a Forward Deployed Engineer. The AI infrastructure field is growing rapidly, and demand for engineers with these capabilities will continue to increase.

Best of luck on your journey!