Kubernetes Multi-Tenancy with vCluster: Virtual Cluster Isolation and Operational Guide

• Introduction
• vCluster Architecture
  • Components
  • Isolation Level Tiers
• Namespace-Based Isolation vs vCluster Comparison
• Installation and Configuration
  • Quick Installation with CLI
  • Production Deployment with Helm
• Syncer Operation Mechanism
  • Synchronization Direction and Resource Types
  • Resource Name Rewriting
  • Label and Annotation Propagation
  • Advanced Sync: Generic Sync
• RBAC and Resource Quota Configuration
  • Virtual Cluster User RBAC
  • Host Cluster Resource Quota
• Network Isolation (NetworkPolicy)
  • Host Cluster Level NetworkPolicy
  • Network Policy within Virtual Cluster
• Monitoring Integration
  • Prometheus Metrics Collection
  • Alert Configuration Recommendations
• Failure Cases and Recovery Procedures
  • Case 1: Pod Mismatch Due to Syncer Synchronization Failure
  • Case 2: Virtual Cluster etcd Data Corruption
  • Case 3: Workload Deployment Failure Due to Resource Quota Exceeded
  • Case 4: Network Isolation Failure Between Virtual Clusters
  • Case 5: Virtual Cluster Upgrade Failure
• Production Operation Checklist
• Conclusion
• References

Introduction

When operating Kubernetes, one challenge you'll inevitably encounter is multi-tenancy. The need to safely isolate workloads from multiple teams, projects, or customers on a single physical cluster while optimizing infrastructure costs becomes increasingly urgent as organizations scale. Traditionally, multi-tenancy has been implemented by combining namespace-based isolation, RBAC policies, and NetworkPolicy, but this approach has inherent limitations: CRD installation permission conflicts, noisy neighbor problems from sharing the API Server, and the inability to upgrade clusters independently per tenant.

vCluster is an open-source project developed by Loft Labs (now vCluster Labs) that addresses these issues by running virtual Kubernetes clusters within a namespace of the host cluster. Each virtual cluster has its own dedicated API Server, Controller Manager, and data store (etcd or SQLite), allowing you to grant tenants complete cluster management authority while still sharing actual compute resources from the host cluster. As a CNCF project, vCluster introduced vNode at KubeCon EU 2025, adding a node-level virtualization isolation layer, and later launched Standalone vCluster to enable independent virtual cluster operation without a host cluster.

This article comprehensively covers vCluster architecture, installation, configuration, the Syncer mechanism, RBAC policies, resource quotas, network isolation, monitoring integration, and real production failure cases with recovery procedures.

vCluster Architecture

The core idea of vCluster is to use one namespace of the host cluster as the execution environment for the virtual cluster. Workloads created inside the virtual cluster are actually scheduled as Pods in the host cluster's namespace through the Syncer component.

Components

vCluster comprises the following main components:

1. Control Plane: Runs a dedicated API Server and Controller Manager for the virtual cluster. By default it uses k3s, but k0s and vanilla k8s (including EKS distro) are also available as options.
2. Data Store: Uses etcd, SQLite, or external databases (PostgreSQL, MySQL) as the data store. The default for k3s-based deployments is the built-in SQLite.
3. Syncer: The core component responsible for resource synchronization between the virtual cluster and the host cluster.
4. CoreDNS: Handles DNS resolution within the virtual cluster. Works in conjunction with the host cluster's DNS to support service discovery.

Isolation Level Tiers

vCluster provides three levels of isolation:

• Shared (Default): The virtual cluster runs within a namespace of the host cluster and shares the host cluster's nodes.
• Private (Dedicated): Compute resources are physically isolated by allocating dedicated nodes for the virtual cluster.
• Standalone (Independent): A virtual cluster that runs independently without a host cluster. This feature was introduced in 2025 and provides complete cluster autonomy.

Namespace-Based Isolation vs vCluster Comparison

The following table compares traditional namespace-based multi-tenancy with the virtual cluster approach using vCluster from various perspectives.

Installation and Configuration

Quick Installation with CLI

The vCluster CLI enables the fastest way to create a virtual cluster:

# Install vCluster CLI
curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64"
chmod +x vcluster
sudo mv vcluster /usr/local/bin/

# Create virtual cluster (k3s-based by default)
vcluster create my-vcluster --namespace team-alpha

# Access virtual cluster (kubeconfig automatically configured)
vcluster connect my-vcluster --namespace team-alpha

# Verify connectivity
kubectl get namespaces
kubectl get nodes

# Disconnect from virtual cluster
vcluster disconnect

# Delete virtual cluster
vcluster delete my-vcluster --namespace team-alpha

Production Deployment with Helm

For production environments, using Helm charts with fine-grained configuration is recommended. Here's an example of a production-ready vcluster.yaml configuration:

# vcluster.yaml - Production configuration example
controlPlane:
  # Use k8s instead of k3s (production recommended)
  distro:
    k8s:
      enabled: true
      apiServer:
        extraArgs:
          - '--audit-log-path=/var/log/kubernetes/audit.log'
          - '--audit-log-maxage=30'
          - '--audit-log-maxbackup=10'
          - '--audit-log-maxsize=100'
      controllerManager:
        extraArgs:
          - '--terminated-pod-gc-threshold=50'

  # Control plane resource limits
  statefulSet:
    resources:
      limits:
        cpu: '2'
        memory: 4Gi
      requests:
        cpu: '500m'
        memory: 1Gi
    persistence:
      size: 20Gi

  # Use external etcd (high availability)
  backingStore:
    etcd:
      deploy:
        enabled: true
        statefulSet:
          resources:
            limits:
              cpu: '1'
              memory: 2Gi
            requests:
              cpu: '200m'
              memory: 512Mi

# Resource synchronization configuration
sync:
  toHost:
    pods:
      enabled: true
    services:
      enabled: true
    configmaps:
      enabled: true
    secrets:
      enabled: true
    endpoints:
      enabled: true
    persistentvolumeclaims:
      enabled: true
    ingresses:
      enabled: true
    storageClasses:
      enabled: false
  fromHost:
    nodes:
      enabled: true
      selector:
        labels:
          team: alpha
    storageClasses:
      enabled: true
    ingressClasses:
      enabled: true

# Networking configuration
networking:
  replicateServices:
    fromHost:
      - from: monitoring/prometheus-server
        to: monitoring/prometheus-server
  resolveDNS:
    - hostname: '*.team-alpha.svc.cluster.local'
      target:
        hostNamespace: team-alpha

# Security policies
policies:
  resourceQuota:
    enabled: true
    quota:
      requests.cpu: '8'
      requests.memory: '16Gi'
      limits.cpu: '16'
      limits.memory: '32Gi'
      pods: '50'
      services: '20'
      persistentvolumeclaims: '10'
  limitRange:
    enabled: true
    default:
      cpu: '500m'
      memory: '512Mi'
    defaultRequest:
      cpu: '100m'
      memory: '128Mi'

The deployment command using Helm chart is as follows:

# Add Helm repository
helm repo add loft https://charts.loft.sh
helm repo update

# Create namespace
kubectl create namespace team-alpha

# Deploy vCluster
helm upgrade --install my-vcluster loft/vcluster \
  --namespace team-alpha \
  --values vcluster.yaml \
  --version 0.24.1

# Check deployment status
kubectl get pods -n team-alpha
kubectl get statefulset -n team-alpha

# Access vCluster
vcluster connect my-vcluster -n team-alpha

Syncer Operation Mechanism

The Syncer is the core engine of vCluster, responsible for resource synchronization between the virtual cluster and the host cluster. Understanding this mechanism precisely is essential for vCluster operations.

Synchronization Direction and Resource Types

Syncer synchronization occurs in two directions:

Virtual → Host (toHost): Resources created in the virtual cluster are actually created in the host cluster's namespace. Pods, Services, ConfigMaps, Secrets, and PVCs are synchronized in this direction. When you create a Deployment in the virtual cluster, the Deployment itself exists only in the virtual cluster's etcd, but the resulting Pods are actually scheduled as Pods in the host cluster's namespace.

Host → Virtual (fromHost): Resources from the host cluster are made available within the virtual cluster. StorageClass, IngressClass, Node information, and PriorityClass are synchronized in this direction.

Resource Name Rewriting

When the Syncer synchronizes resources from the virtual cluster to the host cluster, it rewrites names to prevent name collisions. The default rewriting rule follows the pattern {vcluster-name}-x-{resource-name}-x-{vcluster-namespace}. For example, if you create an nginx Pod in the default namespace of the my-vcluster virtual cluster, it will be created with the name my-vcluster-x-nginx-x-team-alpha in the host cluster.

Label and Annotation Propagation

The Syncer maintains relationships with the virtual cluster by attaching additional labels to resources in the host cluster. Labels like vcluster.loft.sh/managed-by and vcluster.loft.sh/namespace are automatically added and used for garbage collection and resource tracking.

Advanced Sync: Generic Sync

For custom resources (CRDs) beyond the default supported resources, you can use the Generic Sync feature. This allows you to synchronize custom resources such as Cert-Manager's Certificate or Istio's VirtualService from the virtual cluster to the host cluster.

RBAC and Resource Quota Configuration

Virtual Cluster User RBAC

Within the virtual cluster, you can grant tenants cluster-admin permissions. This is only valid within the virtual cluster scope and doesn't impact the security of the host cluster.

# vcluster-tenant-rbac.yaml
# Grant admin permissions to tenants within the virtual cluster
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tenant-admin-binding
subjects:
  - kind: Group
    name: team-alpha-developers
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
---
# Restrict access to vCluster namespace in host cluster
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: vcluster-namespace-viewer
  namespace: team-alpha
rules:
  - apiGroups: ['']
    resources: ['pods', 'services', 'configmaps']
    verbs: ['get', 'list', 'watch']
  - apiGroups: ['']
    resources: ['pods/log']
    verbs: ['get']
  - apiGroups: ['apps']
    resources: ['statefulsets']
    verbs: ['get', 'list', 'watch']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: vcluster-namespace-viewer-binding
  namespace: team-alpha
subjects:
  - kind: Group
    name: team-alpha-developers
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: vcluster-namespace-viewer
  apiGroup: rbac.authorization.k8s.io

Host Cluster Resource Quota

To prevent the virtual cluster from consuming excessive resources from the host cluster, apply a ResourceQuota to the namespace where the virtual cluster runs.

# host-resource-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: vcluster-team-alpha-quota
  namespace: team-alpha
spec:
  hard:
    requests.cpu: '8'
    requests.memory: '16Gi'
    limits.cpu: '16'
    limits.memory: '32Gi'
    pods: '50'
    services: '20'
    services.loadbalancers: '2'
    services.nodeports: '5'
    persistentvolumeclaims: '10'
    requests.storage: '100Gi'
    count/deployments.apps: '30'
    count/statefulsets.apps: '10'
    count/jobs.batch: '20'
    count/cronjobs.batch: '10'
---
apiVersion: v1
kind: LimitRange
metadata:
  name: vcluster-team-alpha-limits
  namespace: team-alpha
spec:
  limits:
    - type: Container
      default:
        cpu: '500m'
        memory: '512Mi'
      defaultRequest:
        cpu: '100m'
        memory: '128Mi'
      max:
        cpu: '4'
        memory: '8Gi'
      min:
        cpu: '50m'
        memory: '64Mi'
    - type: Pod
      max:
        cpu: '8'
        memory: '16Gi'
    - type: PersistentVolumeClaim
      max:
        storage: '50Gi'
      min:
        storage: '1Gi'

Network Isolation (NetworkPolicy)

In vCluster environments, network isolation is implemented at two layers. First, at the host cluster level, you control communication between virtual cluster namespaces, and additionally, you can apply fine-grained policies between workloads within the virtual cluster itself.

Host Cluster Level NetworkPolicy

# host-networkpolicy.yaml
# Block communication between virtual cluster namespaces
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: isolate-vcluster-namespace
  namespace: team-alpha
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Allow communication within the same namespace
    - from:
        - podSelector: {}
    # Allow traffic from Ingress controller
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
          podSelector:
            matchLabels:
              app.kubernetes.io/name: ingress-nginx
    # Allow metric collection from monitoring system
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
          podSelector:
            matchLabels:
              app: prometheus
  egress:
    # Allow DNS queries
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    # Allow communication within the same namespace
    - to:
        - podSelector: {}
    # Allow external internet access (if needed)
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16

This NetworkPolicy isolates the virtual cluster's host namespace, blocking direct communication with other tenant namespaces. Simultaneously, it permits communication with essential infrastructure services such as the Ingress controller, monitoring system, and DNS.

Network Policy within Virtual Cluster

Within the virtual cluster, you can also apply network policies between microservices. NetworkPolicies defined within the virtual cluster are appropriately transformed and applied through the Syncer to the host cluster.

Monitoring Integration

Prometheus Metrics Collection

To configure monitoring for vCluster's control plane and workloads within the virtual cluster, the Prometheus in the host cluster must be able to scrape Pods in the virtual cluster's namespace.

You can collect metrics from the vCluster control plane by adding ServiceMonitor to the Prometheus in the host cluster. This includes metrics from the vCluster API Server, etcd, and Syncer synchronization status, providing comprehensive monitoring of control plane health. In particular, the Syncer's synchronization delay time and error rate are key indicators for determining virtual cluster stability.

Metrics from workloads within the virtual cluster can be collected either by having the host cluster's Prometheus directly scrape Pods in the namespace, or by installing a separate Prometheus within the virtual cluster. In multi-tenant environments, it's good practice to configure per-tenant Grafana dashboards, applying access controls so each team can only see their own workload status.

Alert Configuration Recommendations

In production environments, it is recommended to configure alerts for the following items:

• vCluster control plane Pod restart count exceeds threshold
• Syncer synchronization delay time exceeds 30 seconds
• Virtual cluster namespace resource quota usage exceeds 80%
• etcd data store disk usage exceeds 70%
• vCluster API Server response time increases abnormally

Failure Cases and Recovery Procedures

Case 1: Pod Mismatch Due to Syncer Synchronization Failure

Symptom: You deleted a Deployment in the virtual cluster but orphan Pods remain in the host cluster. The list of Pods from kubectl get pods in the virtual cluster doesn't match the actual Pods in the host cluster.

Cause: This occurs when deletion events are not propagated due to Syncer Pod OOM Kill or network partition.

Recovery Procedure:

# 1. Identify orphan Pods in the host cluster
kubectl get pods -n team-alpha -l vcluster.loft.sh/managed-by=my-vcluster \
  --field-selector=status.phase=Running

# 2. Check if the Pod exists in the virtual cluster
vcluster connect my-vcluster -n team-alpha
kubectl get pods --all-namespaces

# 3. Manually delete orphan Pods that don't exist in the virtual cluster
vcluster disconnect
kubectl delete pod <orphan-pod-name> -n team-alpha --grace-period=30

# 4. Restart the Syncer Pod to restore synchronization state
kubectl rollout restart statefulset my-vcluster -n team-alpha

# 5. Verify synchronization state (review logs)
kubectl logs statefulset/my-vcluster -n team-alpha -c syncer --tail=100

Prevention Measures: Set sufficient resource requests/limits for the Syncer and configure a CronJob that periodically checks for resource consistency between the virtual cluster and host cluster.

Case 2: Virtual Cluster etcd Data Corruption

Symptom: The virtual cluster API Server fails to start, and the etcd container logs show errors like panic: freepages: failed to get all reachable pages or database file is not valid.

Cause: etcd data file becomes corrupted due to disk I/O errors in PersistentVolume, abnormal Pod termination, or storage backend failure.

Recovery Procedure:

1. Scale down the virtual cluster's StatefulSet to replicas: 0.
2. If a backup exists, replace the PVC data with the backup data.
3. If no backup exists, delete the PVC and replace it with a new one, then recreate the virtual cluster. In this case, all metadata within the virtual cluster will be lost, but the actual workload Pods will continue running in the host cluster.
4. Scale the StatefulSet back up to replicas: 1 and verify that it resynchronizes with the existing resources in the host cluster.

Prevention Measures: Configure regular snapshot backups of etcd data, and use a high-availability etcd cluster configuration in production environments.

Case 3: Workload Deployment Failure Due to Resource Quota Exceeded

Symptom: Pod creation fails within the virtual cluster, but instead of a forbidden: exceeded quota error, the Pod remains stuck in Pending state.

Cause: The ResourceQuota in the host cluster has been exceeded, but the error message is not properly propagated to the virtual cluster's events, making it difficult for tenants to identify the root cause.

Recovery Procedure:

1. Check the ResourceQuota usage for the namespace in the host cluster: kubectl describe resourcequota -n team-alpha
2. Clean up unnecessary resources or increase the quota.
3. Enable event synchronization in the vCluster configuration so tenants can also see events from the host cluster.

Prevention Measures: Generate an alert when quota usage exceeds 80%, and configure a separate ResourceQuota within the virtual cluster for pre-emptive limitation.

Case 4: Network Isolation Failure Between Virtual Clusters

Symptom: Workloads from different virtual clusters can communicate directly because they share the same network in the host cluster.

Cause: NetworkPolicy is not configured on the host cluster, or the CNI plugin doesn't support NetworkPolicy (e.g., Flannel with default settings).

Recovery Procedure:

1. Verify that the CNI plugin supports NetworkPolicy. Calico, Cilium, and WeaveNet support NetworkPolicy, while Flannel with default settings does not.
2. Apply the NetworkPolicy presented in the network isolation section to the host namespace of each virtual cluster.
3. Test isolation by creating test Pods to verify that communication to other namespaces is blocked.

Case 5: Virtual Cluster Upgrade Failure

Symptom: After upgrading the vCluster version via Helm, the virtual cluster API Server fails to start or compatibility issues occur with existing workloads.

Cause: API changes between major versions or Syncer protocol changes causing incompatibility.

Recovery Procedure:

1. Always perform an etcd snapshot backup before upgrading.
2. On failure, use helm rollback my-vcluster <previous-revision> -n team-alpha to rollback to the previous version.
3. If issues persist after rollback, restore etcd data from the backup.

Prevention Measures: Test the upgrade in a staging environment first, and always review Breaking Changes in the official upgrade guide. In production environments, upgrade incrementally one step at a time.

Production Operation Checklist

Key items to verify when operating vCluster in a production environment:

Pre-Deployment Checks:

• Verify that the CNI plugin supports NetworkPolicy (Calico, Cilium recommended)
• Verify that ResourceQuota is applied to the virtual cluster namespace in the host cluster
• Verify that vCluster control plane resource requests/limits are properly configured
• Verify that PersistentVolume storage class and reclaim policy are correct
• Verify that the kubeconfig distribution process for the virtual cluster is secure (OIDC integration recommended)

Monitoring During Operation:

• Monitor Syncer synchronization delay time and error rate
• Monitor CPU/memory usage of virtual cluster control plane
• Monitor disk usage and compaction status of etcd data store
• Monitor request delay time and error rate of virtual cluster API Server
• Monitor resource quota usage for the host cluster namespace

Backup and Disaster Recovery:

• Verify that etcd snapshot backup schedule is configured (at least once daily)
• Verify that backup restoration procedure is tested quarterly
• Verify that virtual cluster recreation procedure is automated (IaC)
• Verify that virtual cluster recovery procedure in case of host cluster failure is documented

Security Checks:

• Verify that virtual cluster user authentication/authorization is integrated with central IdP (OIDC, LDAP)
• Verify that Node access permissions in the host cluster are not exposed to virtual cluster tenants
• Verify that Pod Security Standards (PSS) are applied to prevent privileged container execution
• Verify that network isolation between virtual clusters is tested periodically

Upgrade Procedure:

• Perform etcd snapshot backup before vCluster version upgrade
• Test upgrade in staging environment before production deployment
• Verify Syncer synchronization status and workload normal operation after upgrade
• Prepare rollback procedure and commands in advance

Conclusion

vCluster solves the fundamental limitations of Kubernetes multi-tenancy through an abstraction layer of virtual clusters. You can achieve requirements like CRD independence, API Server isolation, and per-tenant cluster management autonomy without adding physical clusters, which namespace-based isolation alone cannot fully satisfy.

However, without a thorough understanding of the Syncer operation mechanism and resource name rewriting, you may encounter unexpected issues during operations. It is recommended to conduct sufficient validation in a staging environment before deploying to production. In particular, network isolation and resource quota configuration must be applied together during the initial setup phase.

References

• vCluster Official Documentation - Virtual Clusters for Kubernetes
• vCluster GitHub Repository
• vCluster Sync Configuration Guide
• vCluster Architecture - Networking
• Kubernetes Multi-Tenancy Working Group