Modelplane Modelplane docs
Version
main v0.1 v0.2 Latest

Build the platform

On this page

This is the platform team’s side of Modelplane. You set up the gateway that fronts your models, give the control plane cloud credentials, and register your first GPU cluster: a hardware profile published as an InferenceClass and an InferenceCluster that offers it.

In the next step, the ML team will create a model deployment that schedules against this capacity without knowing which cluster it runs on.

Prerequisites

  • An AWS account with permissions to create EKS clusters, VPCs, and IAM roles
  • AWS access key ID and secret access key
  • A GCP account with permissions to create GKE clusters, VPCs, and IAM roles
  • A GCP service account JSON key
  • An Azure account with permissions to create AKS clusters and managed identities
  • An Azure service principal JSON with clientId, clientSecret, subscriptionId, and tenantId
  • A Nebius account with permissions to create clusters
  • A Nebius service account JSON key and your project ID

Set up the InferenceGateway

The InferenceGateway installs Traefik Proxy and MetalLB on the control plane. Traefik routes inference traffic to model replicas. MetalLB assigns Traefik’s LoadBalancer service an external IP on kind, which doesn’t have a cloud load balancer. You need one named default per control plane.

If you run the control plane on a cloud cluster with native LoadBalancer support, omit the loadBalancer field.

inference-gateway.yaml
# The InferenceGateway creates a unified, OpenAI-compatible endpoint on the
# control plane cluster. It installs Traefik Proxy and creates a Gateway that
# routes traffic to model replicas on remote inference clusters.
#
# Create one InferenceGateway per control plane. It must be named "default".
#
# For kind or bare-metal clusters, set loadBalancer to MetalLB and configure an
# address pool. For cloud clusters with native LoadBalancer support, omit the
# loadBalancer field entirely.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceGateway
metadata:
  name: default
spec:
  backend: Traefik
  traefik:
    version: "40.2.0"

    # Remove the loadBalancer section if your cluster supports LoadBalancer
    # services natively (e.g. GKE, EKS).
    loadBalancer: MetalLB
    metallb:
      addressPool: "172.18.255.200-172.18.255.250"

Wait until the gateway is ready:

bash
kubectl wait --for=condition=Ready ig/default --timeout=5m

Configure cloud credentials

Give the control plane credentials so it can provision clusters in your cloud account.

Create an AWS credentials file:

ini
[default]
aws_access_key_id = 
aws_secret_access_key =

Create a Kubernetes secret:

bash
kubectl create secret generic aws-creds \
  --from-file=credentials= \
  -n crossplane-system

Apply the ClusterProviderConfig referencing your secret:

clusterproviderconfig-aws.yaml
# Points the AWS provider at the credentials Secret you created. Named default,
# so InferenceClusters with an EKS source use it without further configuration.
apiVersion: aws.m.upbound.io/v1beta1
kind: ClusterProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: credentials

Create a Kubernetes secret:

bash
kubectl create secret generic gcp-creds \
  --from-file=credentials=.json \
  -n crossplane-system

Apply the ClusterProviderConfig, setting projectID to your GCP project:

clusterproviderconfig-gke.yaml
# Points the GCP provider at the credentials Secret you created. Named default,
# so InferenceClusters with a GKE source use it without further configuration.
apiVersion: gcp.m.upbound.io/v1beta1
kind: ClusterProviderConfig
metadata:
  name: default
spec:
  projectID: my-gcp-project  # replace with your GCP project
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: gcp-creds
      key: credentials
bash
curl -fsSL /examples/getting-started/clusterproviderconfig-gke.yaml \
  | sed 's/my-gcp-project//' \
  | kubectl apply -f -

Create a Kubernetes secret from your service principal JSON:

bash
kubectl create secret generic azure-credentials \
  --from-file=credentials.json=.json \
  -n crossplane-system

Apply the ClusterProviderConfig referencing your secret:

clusterproviderconfig-azure.yaml
# Points the Azure providers at the credentials Secret you created. Named
# default, so InferenceClusters with an AKS source use it without further
# configuration. The Secret carries a service principal JSON with
# clientId, clientSecret, subscriptionId, and tenantId.
apiVersion: azure.m.upbound.io/v1beta1
kind: ClusterProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: azure-credentials
      key: credentials.json

Create a Kubernetes secret from your service account JSON:

bash
kubectl create secret generic nebius-credentials \
  --from-file=credentials.json=.json \
  -n crossplane-system

Apply the ClusterProviderConfig, setting projectID to your Nebius project:

clusterproviderconfig-nebius.yaml
# Points the Nebius provider at the credentials Secret you created. Named
# default, so InferenceClusters with a Nebius source use it without further
# configuration - Modelplane also reuses its credentials Secret to
# authenticate to the clusters it provisions.
apiVersion: nebius.m.upbound.io/v1beta1
kind: ClusterProviderConfig
metadata:
  name: default
spec:
  identity:
    type: ServiceAccount
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: nebius-credentials
      key: credentials.json
  projectID: project-e00example
bash
curl -fsSL /examples/getting-started/clusterproviderconfig-nebius.yaml \
  | sed 's/project-e00example//' \
  | kubectl apply -f -

Publish hardware and register the cluster

The InferenceClass describes a hardware profile and how to provision it. The InferenceCluster registers a cluster that offers it. Apply both:

platform.yaml
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: l4-1x-g6
spec:
  description: "EKS g6.xlarge, 1x NVIDIA L4"
  provisioning:
    provider: EKS
    eks:
      instanceType: g6.xlarge
      diskSizeGb: 50
      accelerator:
        type: nvidia-l4
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
    capacity:
      memory: { value: "23034Mi" }   # L4's real reported VRAM (not the nominal 24GB)
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-us-east
  labels:
    modelplane.ai/region: us-east
spec:
  cluster:
    source: EKS
    eks:
      region: us-east-1
  nodePools:
  - name: gpu-l4
    className: l4-1x-g6
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
    zones:
    - us-east-1b

Modelplane provisions the cluster. This takes about 15 minutes:

bash
kubectl wait --for=condition=Ready ic/eks-us-east --timeout=20m

Apply the manifest, setting the cluster’s project to your GCP project:

platform.yaml
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: gke-l4-1x-g2
spec:
  description: "GKE g2-standard-8, 1x NVIDIA L4"
  provisioning:
    provider: GKE
    gke:
      machineType: g2-standard-8
      diskSizeGb: 100
      accelerator:
        type: nvidia-l4
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
    capacity:
      memory: { value: "23034Mi" }   # L4's real reported VRAM (not the nominal 24GB)
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: starter
  labels:
    modelplane.ai/region: us-central
spec:
  cluster:
    source: GKE
    gke:
      project: my-gcp-project
      region: us-central1
  nodePools:
  - name: gpu-l4
    className: gke-l4-1x-g2
    nodeCount: 1
    minNodeCount: 0
    maxNodeCount: 2
    zones:
    - us-central1-a
bash
curl -fsSL /examples/getting-started/gke/platform.yaml \
  | sed 's/my-gcp-project//' \
  | kubectl apply -f -

Modelplane provisions the cluster. This takes about 15 minutes:

bash
kubectl wait --for=condition=Ready ic/starter --timeout=20m
platform.yaml
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: a10-1x
spec:
  description: "AKS Standard_NV36ads_A10_v5, 1x NVIDIA A10 24GB"
  provisioning:
    provider: AKS
    aks:
      # verify: Azure has no L4 SKU; NVadsA10v5 is the small-GPU option (>=20Gi).
      # Confirm it runs the DRA / GPU-operator path in your subscription.
      vmSize: Standard_NV36ads_A10_v5
      diskSizeGb: 100
      accelerator:
        type: nvidia-a10
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.6.0" }
    capacity:
      memory: { value: "24564Mi" }   # verify: A10 reported VRAM (~24GB)
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: aks-westeurope
  labels:
    modelplane.ai/region: westeurope
spec:
  cluster:
    source: AKS
    aks:
      location: westeurope
  nodePools:
  - name: gpua10
    className: a10-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

Modelplane provisions the cluster. This takes about 15 minutes:

bash
kubectl wait --for=condition=Ready ic/aks-westeurope --timeout=20m
platform.yaml
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: l40s-1x
spec:
  description: "Nebius gpu-l40s-a, 1x NVIDIA L40S 48GB"
  provisioning:
    provider: Nebius
    nebius:
      # gpu-l40s-a (L40S PCIe, Intel Ice Lake) is available in eu-north1.
      platform: gpu-l40s-a
      preset: 1gpu-8vcpu-32gb
      diskSizeGb: 100
      driversPreset: cuda13.0
      accelerator:
        type: nvidia-l40s
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
      cudaComputeCapability: { version: "8.9.0" }
    capacity:
      memory: { value: "46068Mi" }   # L40S reported VRAM (48GB nominal)
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: nebius-eu-north
  labels:
    modelplane.ai/region: eu-north
spec:
  cluster:
    source: Nebius
    nebius: {}
  nodePools:
  - name: gpu-l40s
    className: l40s-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

Modelplane provisions the cluster. This takes about 15 minutes:

bash
kubectl wait --for=condition=Ready ic/nebius-eu-north --timeout=20m
Note

Modelplane is reconciling the infrastructure against the source of truth, the manifest you just applied.

While you wait, Modelplane is creating the cloud cluster and its GPU node pool, then installing the inference stack with LeaderWorkerSet for multi-node serving, llm-d for inference-aware routing, Envoy Gateway for traffic management, and the storage class for model weights. This is the same reconciliation loop Crossplane uses to configure other infrastructure, extended to the inference layer.

Once the cluster is Ready the ML team can deploy a model on it.

Note
A cloud GPU cluster costs money while it runs. To stop the tour and resume later, follow Clean up.

Next step

Now that the platform is provisioned, the ML team can deploy a model by describing what the model needs, not the infrastructure.