OME

OME API

Mon, 01 Jan 0001 00:00:00 +0000

Package v1beta1 contains API Schema definitions for the serving v1beta1 API group

Resource Types

`BaseModel`

Appears in:

Labels and Annotations

Mon, 01 Jan 0001 00:00:00 +0000

This document serves as a reference of the various labels and annotations used throughout OME.

Annotations

InferenceService Annotations

These annotations are used to configure InferenceService behavior:

Base Model

Tue, 14 Mar 2023 00:00:00 +0000

What is a Base Model?

A Base Model in OME is a Kubernetes resource that represents a foundation AI model (like GPT, Llama, or Mistral) that you want to use for inference workloads. Think of it as a blueprint that tells OME where to find your model, how to download it, and where to store it on your cluster nodes.

Contributing to OME

Tue, 14 Mar 2023 00:00:00 +0000

Thank you for your interest in contributing to OME! This repository is open to everyone and welcomes all kinds of contributions, no matter how small or large.

Ways to Contribute

There are several ways you can contribute to the project:

Bug Reports: Identify and report issues or bugs
Feature Requests: Suggest new features or improvements
Code Contributions: Submit bug fixes or implement new features
Documentation: Improve documentation and guides
Testing: Help with testing and quality assurance

Development Environment Setup

Prerequisites

Before you begin, you’ll need to install these tools:

Deploy a Simple Inference Service

Tue, 14 Mar 2023 00:00:00 +0000

This page shows you how to deploy a simple inference service using OME. You’ll learn how to create an InferenceService that serves a pre-trained model for real-time inference using SGLang and OpenAI-compatible APIs.

Before you begin

You need to have the following:

A Kubernetes cluster with OME installed
kubectl configured to communicate with your cluster
GPU nodes available in your cluster (A100, H100, H200, or B4)
Access to OME container registry (ghcr.io/sgl-project/)

Step 1: Verify prerequisites

Check that OME is installed and running:

Model Agent Administration

Tue, 14 Mar 2023 00:00:00 +0000

The Model Agent is the core component responsible for downloading, managing, and distributing models across your OME cluster. This guide provides comprehensive information for cluster administrators who need to configure, monitor, and troubleshoot the Model Agent in production environments.

Architecture Overview

DaemonSet Deployment

The Model Agent is deployed as a Kubernetes DaemonSet, ensuring it runs on every node in your cluster. This distributed architecture provides several benefits:

Controller Configuration

Mon, 01 Jan 0001 00:00:00 +0000

The OME controller manager (ome-manager) is configured through command-line flags passed to its container. This page documents those flags and how to change them on a running cluster.

Setting flags

The flags are parsed once at startup, so a change takes effect when the controller pod restarts. No image rebuild is required.

Runtime Revisions and Pinning

Mon, 01 Jan 0001 00:00:00 +0000

By default an InferenceService always renders its pods from the live ServingRuntime it references: whenever the runtime changes, the next reconcile picks up the new spec. Runtime pinning lets you decouple an InferenceService from live runtime changes by pinning it to an immutable snapshot of the runtime, so updates roll out only when you ask for them.

Snapshots are stored as Kubernetes ControllerRevision objects in the OME namespace, are content-addressed (deduplicated by hash), and are garbage-collected on a configurable schedule.

Run Performance Benchmarks

Tue, 14 Mar 2023 00:00:00 +0000

This page shows you how to run performance benchmarks on your inference services using OME’s BenchmarkJob. You’ll learn how to test different traffic scenarios, measure performance metrics, and store results for analysis.

Before you begin

You need to have the following:

A Kubernetes cluster with OME installed
kubectl configured to communicate with your cluster
An InferenceService deployed and ready
Access to storage for benchmark results (OCI Object Storage or PVC)
OME benchmark tool image available

Step 1: Verify prerequisites

Check that your inference service is running:

Fine Tuned Weight

Mon, 01 Jan 0001 00:00:00 +0000

What is a Fine Tuned Weight?

A FineTunedWeight in OME is a Kubernetes resource that represents a set of weights that were fine-tuned from an existing Base Model. Rather than describing a full, standalone model, it points back to a base model and carries only the artifacts and metadata specific to the fine-tuning - for example a LoRA adapter, an added adapter module, or a distilled variant.

Serving Runtime

Tue, 14 Mar 2023 00:00:00 +0000

The only difference between the two is that one is namespace-scoped and the other is cluster-scoped.

A ClusterServingRuntime defines the templates for Pods that can serve one or more particular model. Each ClusterServingRuntime defines key information such as the container image of the runtime and a list of the models that the runtime supports. Other configuration settings for the runtime can be conveyed through environment variables in the container specification.

Inference Service

Tue, 14 Mar 2023 00:00:00 +0000

What is an InferenceService?

An InferenceService is the central Kubernetes resource in OME that orchestrates the complete lifecycle of model serving. It acts as a declarative specification that describes how you want your AI models deployed, scaled, and served across your cluster.

Think of InferenceService as the “deployment blueprint” for your AI workloads. It brings together models (defined by BaseModel/ClusterBaseModel), runtimes (defined by ServingRuntime/ClusterServingRuntime), and infrastructure configuration to create a complete serving solution.

Benchmark

Tue, 14 Mar 2023 00:00:00 +0000

A BenchmarkJob is a resource in OME that automates the performance benchmarking of inference service or OCI Generative AI Service endpoints. It allows you to evaluate model serving performance under various traffic patterns and load conditions.

BenchmarkJob uses genai-bench, a comprehensive benchmarking tool for evaluating generative AI model serving systems. For detailed information about genai-bench features and capabilities, refer to the official genai-bench documentation.

Core Components

A BenchmarkJob consists of several key components:

Ingress and External Access

Fri, 21 Jun 2024 00:00:00 +0000

What is Ingress in OME?

Ingress in OME provides external access to your AI inference services running inside the Kubernetes cluster. When you deploy an InferenceService, OME automatically creates the appropriate ingress resources based on your deployment mode and cluster configuration, allowing external clients to make API calls to your models.

Ingress Administration

Mon, 01 Jan 0001 00:00:00 +0000

OME (Open Model Engine) provides flexible ingress configuration to support various Kubernetes ingress controllers and deployment scenarios. This guide covers the configuration options available to cluster administrators.

Supported Ingress Controllers

OME supports the following ingress solutions:

Standard Kubernetes Ingress Controllers

NGINX Ingress Controller - Installation Guide
Traefik - Installation Guide
Kong Ingress Controller - Installation Guide
HAProxy Ingress - Installation Guide
Contour - Installation Guide

Service Mesh Ingress

Istio (VirtualService + Gateway) - Installation Guide

Gateway API

Gateway API Controllers - Installation Guide

OME Ingress Configuration

OME ingress behavior is configured through the inferenceservice-config ConfigMap in the OME controller namespace.

Search Results

Mon, 01 Jan 0001 00:00:00 +0000