Inference Service

What is an InferenceService?

An InferenceService is the central Kubernetes resource in OME that orchestrates the complete lifecycle of model serving. It acts as a declarative specification that describes how you want your AI models deployed, scaled, and served across your cluster.

Think of InferenceService as the “deployment blueprint” for your AI workloads. It brings together models (defined by BaseModel/ClusterBaseModel), runtimes (defined by ServingRuntime/ClusterServingRuntime), and infrastructure configuration to create a complete serving solution.

Architecture Overview

OME uses a component-based architecture where InferenceService can be composed of multiple specialized components:

• Model: References the AI model to serve (BaseModel/ClusterBaseModel)
• Runtime: References the serving runtime environment (ServingRuntime/ClusterServingRuntime)
• Engine: Main inference component that processes requests
• Decoder: Optional component for disaggregated serving (prefill-decode separation)
• Router: Optional component for request routing and load balancing

New vs Deprecated Architecture

apiVersion: ome.io/v1beta1
kind: InferenceService
spec:
  model:
    name: llama-3-70b-instruct
  runtime:
    name: vllm-text-generation
  engine:
    minReplicas: 1
    maxReplicas: 3
    resources:
      requests:
        nvidia.com/gpu: "1"

Top-Level Model Reference (spec.model)

The current way to select a model is the top-level spec.model field, a ModelRef that points at a BaseModel or ClusterBaseModel. Because the reference lives at the top level (not inside a component), the model is managed independently of the serving configuration and is shared by the Engine, Decoder, and Router.

spec:
  model:
    name: llama-3-70b-instruct
    kind: ClusterBaseModel      # or BaseModel for a namespaced model
    fineTunedWeights:
      - my-lora-adapter          # optional, applied on top of the base model

Component Types

Engine Component

The Engine is the primary inference component that processes model requests. It handles model loading, inference execution, and response generation.

spec:
  engine:
    # Pod-level configuration
    serviceAccountName: custom-sa
    nodeSelector:
      accelerator: nvidia-a100

    # Component configuration
    minReplicas: 1
    maxReplicas: 10
    scaleMetric: cpu
    scaleTarget: 70

    # Container configuration
    runner:
      image: custom-vllm:latest
      resources:
        requests:
          nvidia.com/gpu: "2"
        limits:
          nvidia.com/gpu: "2"
      env:
        - name: CUDA_VISIBLE_DEVICES
          value: "0,1"

Decoder Component

The Decoder is used for disaggregated serving architectures where the prefill (prompt processing) and decode (token generation) phases are separated for better resource utilization.

spec:
  decoder:
    minReplicas: 2
    maxReplicas: 8
    runner:
      resources:
        requests:
          cpu: "4"
          memory: "8Gi"

Router Component

The Router handles request routing, cache awareness load balancing, or prefill and decode disaggregation load balancing.

spec:
  router:
    minReplicas: 1
    maxReplicas: 3
    config:
      routing_strategy: "round_robin"
      health_check_interval: "30s"
    runner:
      resources:
        requests:
          cpu: "1"
          memory: "2Gi"

Deployment Modes

OME automatically selects the optimal deployment mode based on your configuration:

Raw Deployment Mode (Default)

Uses standard Kubernetes Deployments with full control over pod lifecycle and scaling.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-chat
spec:
  model:
    name: llama-3-70b-instruct
  engine:
    minReplicas: 2
    maxReplicas: 10

This deployment mode offers direct Kubernetes management with standard HPA-based autoscaling, no cold starts, and is ideal for stable, predictable workloads.

Serverless Mode

Leverages Knative Serving for automatic scaling including scale-to-zero capabilities.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-chat
spec:
  model:
    name: llama-3-70b-instruct
  engine:
    minReplicas: 0  # Enables scale-to-zero
    maxReplicas: 10
    scaleTarget: 10  # Concurrent requests per pod

This deployment mode leverages Knative Serving for request-based autoscaling, scale-to-zero when idle, and is ideal for variable workloads and cost-sensitive environments.

⚠️ WARNING: This deployment mode leverages Knative Serving for request-based autoscaling, scale-to-zero when idle, and is ideal for variable workloads and cost-sensitive environments; however, it may introduce additional startup latency for large language models due to cold starts and model loading time.

Multi-Node Mode

Enables distributed model serving across multiple nodes using LeaderWorkerSet or Ray clusters.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: deepseek-chat
spec:
  model:
    name: deepseek-r1  # Large model requiring multiple GPUs
  engine:
    minReplicas: 1
    maxReplicas: 2
    # Worker node configuration
    worker:
      size: 1  # Number of worker nodes

This deployment mode enables distributed inference using LeaderWorkerSet or Ray, with support for multi-GPU and multi-node setups, and is optimized for large language models through automatic coordination between nodes

⚠️ WARNING: Multi-node configurations typically require high-performance networking such as RoCE or InfiniBand, and performance may vary depending on the underlying network topology and hardware provided by different cloud vendors.

Disaggregated Serving (Prefill-Decode)

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: deepseek-ep-disaggregated
spec:
  model:
    name: deepseek-r1

  # Router handles request routing and load balancing for prefill-decode disaggregation
  router:
    minReplicas: 1
    maxReplicas: 3

  # Engine handles prefill phase
  engine:
    minReplicas: 1
    maxReplicas: 3

  # Decoder handles token generation
  decoder:
    minReplicas: 2
    maxReplicas: 8

Multi-Node Serving

When a model is too large to fit on a single node, the Engine (and, for disaggregated serving, the Decoder) can be spread across multiple nodes using a leader/worker topology. OME renders this topology as a LeaderWorkerSet, which schedules the leader pod and its worker pods as a single co-scheduled group.

When Multi-Node Mode is Selected

OME derives the deployment mode from the component spec rather than requiring you to name it explicitly. A component runs in Multi-Node mode as soon as it defines a leader or worker block:

• If engine.leader or engine.worker is set, the Engine is deployed as Multi-Node.
• If decoder.leader or decoder.worker is set, the Decoder is deployed as Multi-Node.
• Otherwise the component falls back to Raw Deployment (or Serverless when minReplicas: 0).

You can also force a specific distributed backend with the ome.io/deploymentMode annotation (for example MultiNode or MultiNodeRayVLLM); when present, the annotation takes precedence over the inferred mode.

Note: The decoder component only supports Raw Deployment or Multi-Node. When a decoder is present, the engine is never placed in Serverless mode.

Leader and Worker Specs

leader and worker are available on both EngineSpec and DecoderSpec:

LeaderSpec and WorkerSpec both embed a full PodSpec and an optional runner container override, so you can tune image, resources, environment, and scheduling separately for the leader and the workers. WorkerSpec adds one extra field:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: deepseek-r1-multinode
spec:
  model:
    name: deepseek-r1
  runtime:
    name: srt-multi-node-deepseek-r1-rdma
  engine:
    minReplicas: 1
    maxReplicas: 1
    # Leader coordinates distributed inference
    leader:
      runner:
        resources:
          requests:
            nvidia.com/gpu: "8"
          limits:
            nvidia.com/gpu: "8"
    # Worker nodes perform distributed processing directed by the leader
    worker:
      size: 1   # one worker pod in addition to the leader (group size = 2)
      runner:
        resources:
          requests:
            nvidia.com/gpu: "8"
          limits:
            nvidia.com/gpu: "8"

Here minReplicas/maxReplicas scale the number of leader/worker groups (each a complete LeaderWorkerSet replica), while worker.size controls how many worker pods sit inside a single group.

⚠️ WARNING: Multi-node configurations typically require high-performance networking such as RoCE or InfiniBand. Performance depends on the underlying network topology and hardware provided by different cloud vendors.

Accelerator Selection

OME can select the accelerator (GPU class) for an InferenceService declaratively instead of requiring hard-coded nodeSelector and resource values. Selection is configured through spec.acceleratorSelector and can be overridden per component.

spec.acceleratorSelector

Selection Policy

policy chooses among the accelerators that satisfy the constraints:

Accelerator Constraints

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-70b
spec:
  model:
    name: llama-3-3-70b-instruct
  acceleratorSelector:
    policy: Cheapest
    constraints:
      minMemory: 80                 # at least 80 GB per accelerator
      architectureFamilies:
        - nvidia-hopper
        - nvidia-ampere
      preferredPrecisions:
        - fp8
        - fp16
  engine:
    minReplicas: 1
    maxReplicas: 3

The accelerator that OME resolves is reported back in the component status under status.components.<component>.selectedAccelerator, including the class name, the reason it was chosen, and the nodeSelector/resource requests that were applied.

Per-Component Accelerator Override

Both EngineSpec and DecoderSpec expose an acceleratorOverride field of the same AcceleratorSelector type. When set, it overrides the top-level spec.acceleratorSelector for that component only. This is useful for disaggregated serving, where the prefill (engine) and decode (decoder) phases may benefit from different hardware.

spec:
  model:
    name: deepseek-r1
  # Cluster-wide default for this service
  acceleratorSelector:
    policy: MostCapable
  engine:
    minReplicas: 1
    maxReplicas: 3
    # Engine (prefill) overrides to a memory-optimized class
    acceleratorOverride:
      constraints:
        minMemory: 141
        architectureFamilies:
          - nvidia-hopper
  decoder:
    minReplicas: 2
    maxReplicas: 8
    # Decoder keeps the service-level selector (MostCapable)

Specification Reference

ModelRef Specification

ServingRuntimeRef Specification

Component Configuration

All components (Engine, Decoder, Router) share this common configuration structure:

RunnerSpec Configuration

KEDA Autoscaling

By default, Raw Deployment components scale with the Kubernetes Horizontal Pod Autoscaler (HPA) driven by the scaleMetric/scaleTarget fields (CPU, memory, concurrency, or RPS). KEDA (Kubernetes Event-driven Autoscaling) is an alternative that lets you scale on custom, application-level signals pulled from Prometheus — for example time-per-output-token, request latency, or queue depth — rather than the built-in resource metrics.

Reach for KEDA when:

• HPA’s CPU/memory/concurrency metrics do not correlate well with your model’s real load.
• You want to scale on a serving-specific Prometheus metric (e.g. sglang_time_per_output_token_seconds).
• Your Prometheus endpoint requires authentication (Grafana Cloud, mTLS, bearer tokens).

KEDA is configured either at the service level via spec.kedaConfig or per component via the component’s own kedaConfig. Set enableKeda: true, point promServerAddress at your Prometheus endpoint, supply a customPromQuery (use %s where the InferenceService name should be substituted), and define scalingThreshold with a scalingOperator. For authenticated endpoints, reference a KEDA TriggerAuthentication via authenticationRef and set authModes. See the fully worked Grafana Cloud example below.

ScalerAuthenticationRef Specification

Example: KEDA with Grafana Cloud Authentication

When using Grafana Cloud or other authenticated Prometheus endpoints, you need to create a TriggerAuthentication resource and reference it in your InferenceService:

# 1. Create a secret with Grafana Cloud credentials
apiVersion: v1
kind: Secret
metadata:
  name: grafana-cloud-auth
  namespace: my-namespace
type: Opaque
stringData:
  username: "123456"  # Grafana Cloud instance ID
  password: "glc_xxx" # Grafana Cloud API token with metrics:read scope
---
# 2. Create a TriggerAuthentication resource
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: grafana-cloud-prometheus-auth
  namespace: my-namespace
spec:
  secretTargetRef:
    - parameter: username
      name: grafana-cloud-auth
      key: username
    - parameter: password
      name: grafana-cloud-auth
      key: password
---
# 3. Reference it in InferenceService
apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: my-model
  namespace: my-namespace
  annotations:
    ome.io/autoscalerClass: keda
spec:
  model:
    name: llama-3-70b-instruct
  engine:
    minReplicas: 1
    maxReplicas: 7
    kedaConfig:
      enableKeda: true
      promServerAddress: "https://prometheus-prod-39-prod-eu-north-0.grafana.net/api/prom"
      authenticationRef:
        name: grafana-cloud-prometheus-auth
        kind: TriggerAuthentication
      authModes: "basic"
      customPromQuery: |
        histogram_quantile(0.5,
          sum by(le) (
            rate(sglang_time_per_output_token_seconds_bucket{ome_io_inferenceservice="%s"}[5m])
          )
        )
      scalingThreshold: "0.07"
      scalingOperator: "GreaterThanOrEqual"

Status and Monitoring

InferenceService Status

The InferenceService status provides comprehensive information about the deployment state:

status:
  url: "http://llama-chat.default.example.com"
  address:
    url: "http://llama-chat.default.svc.cluster.local"
  conditions:
    - type: Ready
      status: "True"
      lastTransitionTime: "2024-01-15T10:30:00Z"
    - type: IngressReady
      status: "True"
      lastTransitionTime: "2024-01-15T10:25:00Z"
  components:
    engine:
      url: "http://llama-chat-engine.default.example.com"
      latestReadyRevision: "llama-chat-engine-00001"
      latestCreatedRevision: "llama-chat-engine-00001"
      traffic:
        - revisionName: "llama-chat-engine-00001"
          percent: 100
          latestRevision: true
    router:
      url: "http://llama-chat-router.default.example.com"
      latestReadyRevision: "llama-chat-router-00001"
  modelStatus:
    transitionStatus: "UpToDate"
    modelRevisionStates:
      activeModelState: "Loaded"
      targetModelState: "Loaded"

Condition Types

Model Status and Troubleshooting

status.modelStatus is the primary place to look when an InferenceService is stuck or not becoming Ready. It has three parts: an overall transitionStatus, the per-revision modelRevisionStates, and — when something goes wrong — a lastFailureInfo block.

transitionStatus

transitionStatus tells you whether the serving endpoints reflect the current spec or are still converging:

modelState

modelRevisionStates reports activeModelState (the model currently serving) and targetModelState (the model being rolled out). Each uses these values:

lastFailureInfo

When transitionStatus is BlockedByFailedLoad or InvalidSpec (or a modelState is FailedToLoad), lastFailureInfo explains why:

Common reason values and what they point to:

Debugging Workflow

1. Read transitionStatus. UpToDate means the model layer is healthy — if the service still is not Ready, the problem is elsewhere (ingress, networking, or a component readiness condition).
2. If it is InProgress, the target model is still loading; watch modelRevisionStates.targetModelState move toward Loaded.
3. If it is BlockedByFailedLoad or InvalidSpec, read lastFailureInfo.reason and lastFailureInfo.message, then act on the table above.

# Inspect the full model status block
kubectl get inferenceservice llama-chat -o jsonpath='{.status.modelStatus}' | jq

# Or view it inline
kubectl get inferenceservice llama-chat -o yaml | grep -A 20 "modelStatus:"

Runtime Pinning

By default spec.runtime.autoSync is true, so OME re-renders each component’s pod spec from the live ServingRuntime on every reconcile. Setting spec.runtime.autoSync: false pins the InferenceService to a ControllerRevision snapshot of the runtime (surfaced as status.pinnedRevisionName), so later edits to the runtime do not roll out until you opt in. You then roll forward by bumping the ome.io/runtime-sync annotation or by setting spec.runtime.revision to a specific snapshot.

For the complete pinning, roll-forward, and rollback workflow, see Runtime Revisions.

Deployment Mode Selection

Choose the appropriate deployment mode based on your requirements:

Best Practices

Resource Management

1. GPU Allocation: Always specify GPU resources explicitly

runner:
  resources:
    requests:
      nvidia.com/gpu: "1"
    limits:
      nvidia.com/gpu: "1"

2. Memory Sizing: Allow 2-4x model size for memory

runner:
  resources:
    requests:
      memory: "32Gi"  # For 8B parameter model

3. CPU Allocation: Provide adequate CPU for preprocessing

runner:
  resources:
    requests:
      cpu: "4"

Scaling Configuration

1. Set Appropriate Limits:

engine:
  minReplicas: 1     # Prevent scale-to-zero for latency
  maxReplicas: 10    # Control costs
  scaleTarget: 70    # 70% CPU utilization target

2. Use KEDA for Custom Metrics:

kedaConfig:
  enableKeda: true
  customPromQuery: "avg_over_time(vllm:request_latency_seconds{service='%s'}[5m])"
  scalingThreshold: "0.5"  # 500ms latency threshold

Troubleshooting

1. Check Component Status:

kubectl get inferenceservice llama-chat -o yaml
kubectl describe inferenceservice llama-chat

2. Monitor Pod Logs:

kubectl logs -l serving.ome.io/inferenceservice=llama-chat

3. Check Resource Usage:

kubectl top pods -l serving.ome.io/inferenceservice=llama-chat