Version

Scale the platform

On this page

You have one small-GPU cluster with a running model. In this guide, you’ll grow the fleet with larger-GPU capacity so the ML team has more to schedule against.

Provisioning takes about 10 to 15 minutes.

Register more clusters

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: l40s-1x-g6e
spec:
  description: "EKS g6e.xlarge, 1x NVIDIA L40S"
  provisioning:
    provider: EKS
    eks:
      instanceType: g6e.xlarge
      diskSizeGb: 100
      accelerator:
        type: nvidia-l40s
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
    capacity:
      memory: { value: "46068Mi" }
---
# g6e.xlarge is available in us-east-1, us-west-2, and eu-central-1.
# eu-west-1 does NOT have g6e.xlarge.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-us-west
  labels:
    modelplane.ai/region: us-west
spec:
  cluster:
    source: EKS
    eks:
      region: us-west-2
  nodePools:
  - name: gpu-l40s
    className: l40s-1x-g6e
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
    zones:
    - us-west-2a
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-eu-central
  labels:
    modelplane.ai/region: eu-central
spec:
  cluster:
    source: EKS
    eks:
      region: eu-central-1
  nodePools:
  - name: gpu-l40s
    className: l40s-1x-g6e
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
    zones:
    - eu-central-1a

Note

g6e.xlarge runs ~$2/hr on demand. Two of them plus the L4 from earlier is a few dollars for this tour. Clean up when you’re done (see Clean up).

Register two more clusters with a bigger hardware class: A100 (40 GB) in us-west and us-east. Apply the manifest, setting each cluster’s project to your GCP project:

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: gke-a100-40-1x
spec:
  description: "GKE a2-highgpu-1g, 1x NVIDIA A100 40GB"
  provisioning:
    provider: GKE
    gke:
      machineType: a2-highgpu-1g
      diskSizeGb: 200
      accelerator:
        type: nvidia-tesla-a100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.0.0" }
    capacity:
      # A100 40GB real reported VRAM. Keep the selector at >= 35Gi (not >= 40Gi)
      # so it reliably clears the L4 (24Gi) without hitting the boundary.
      memory: { value: "40960Mi" }
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gpu-us-west
  labels:
    modelplane.ai/region: us-west
spec:
  cluster:
    source: GKE
    gke:
      project: my-gcp-project
      region: us-west1
  nodePools:
  - name: gpu-a100
    className: gke-a100-40-1x
    nodeCount: 1
    minNodeCount: 0
    maxNodeCount: 2
    zones:
    - us-west1-b
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gpu-us-east
  labels:
    modelplane.ai/region: us-east
spec:
  cluster:
    source: GKE
    gke:
      project: my-gcp-project
      region: us-east1
  nodePools:
  - name: gpu-a100
    className: gke-a100-40-1x
    nodeCount: 1
    minNodeCount: 0
    maxNodeCount: 2
    zones:
    - us-east1-b

curl -fsSL /examples/getting-started/gke/platform-scale.yaml \
  | sed 's/my-gcp-project//g' \
  | kubectl apply -f -

Note

a2-highgpu-1g runs ~$3.50/hr on demand. Two of them plus the L4 from earlier is a few dollars for this tour. Clean up when you’re done (see Clean up).

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: a100-1x
spec:
  description: "AKS Standard_NC24ads_A100_v4, 1x NVIDIA A100 80GB"
  provisioning:
    provider: AKS
    aks:
      vmSize: Standard_NC24ads_A100_v4
      diskSizeGb: 100
      accelerator:
        type: nvidia-a100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.0.0" }
    capacity:
      memory: { value: "81920Mi" }
---
# verify: Standard_NC24ads_A100_v4 quota/availability in eastus and southcentralus.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: aks-eastus
  labels:
    modelplane.ai/region: eastus
spec:
  cluster:
    source: AKS
    aks:
      location: eastus
  nodePools:
  - name: gpua100
    className: a100-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: aks-southcentralus
  labels:
    modelplane.ai/region: southcentralus
spec:
  cluster:
    source: AKS
    aks:
      location: southcentralus
  nodePools:
  - name: gpua100
    className: a100-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

Note

Standard_NC24ads_A100_v4 runs ~$3.70/hr on demand. Two of them plus the A10 from earlier is a few dollars for this tour. Clean up when you’re done (see Clean up).

Nebius projects are bound to one region, so you grow the fleet by GPU tier rather than geography. Register a bigger H100 (80 GB) cluster in the same region:

apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: h100-1x
spec:
  description: "Nebius gpu-h100-sxm, 1x NVIDIA H100 80GB"
  provisioning:
    provider: Nebius
    nebius:
      platform: gpu-h100-sxm
      preset: 1gpu-16vcpu-200gb
      diskSizeGb: 100
      driversPreset: cuda13.0
      accelerator:
        type: nvidia-h100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Hopper }
      cudaComputeCapability: { version: "9.0.0" }
    capacity:
      memory: { value: "81559Mi" }   # H100's real reported VRAM (not the nominal 80GB)
---
# Nebius projects are bound to one region, so this cluster runs in the same
# region as the base cluster. The fleet scales by GPU tier (L40S -> H100), not
# geography. To add a region, create another project + ClusterProviderConfig.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: nebius-eu-north-h100
  labels:
    modelplane.ai/region: eu-north
    modelplane.ai/tier: h100
spec:
  cluster:
    source: Nebius
    nebius: {}
  nodePools:
  - name: gpu-h100
    className: h100-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

Note

The H100 cluster costs more per hour than the L40S from earlier. Clean up when you’re done (see Clean up).

Modelplane provisions the new clusters in parallel:

kubectl wait --for=condition=Ready ic --all --timeout=20m

Your model keeps running

Growing the fleet doesn’t disturb anything already deployed. qwen-demo stays on its original cluster and the two new clusters add capacity the moment they’re Ready with no interruption for the ML team. A replica only moves if its deployment changes in a way that no longer fits where it runs.

Next step

The fleet has grown with larger-GPU capacity. The ML team is next. Scale the model to serve it across the fleet behind a single endpoint.