Concepts on OME

Base Model

Tue, 14 Mar 2023 00:00:00 +0000

What is a Base Model?

A Base Model in OME is a Kubernetes resource that represents a foundation AI model (like GPT, Llama, or Mistral) that you want to use for inference workloads. Think of it as a blueprint that tells OME where to find your model, how to download it, and where to store it on your cluster nodes.

Runtime Revisions and Pinning

Mon, 01 Jan 0001 00:00:00 +0000

By default an InferenceService always renders its pods from the live ServingRuntime it references: whenever the runtime changes, the next reconcile picks up the new spec. Runtime pinning lets you decouple an InferenceService from live runtime changes by pinning it to an immutable snapshot of the runtime, so updates roll out only when you ask for them.

Snapshots are stored as Kubernetes ControllerRevision objects in the OME namespace, are content-addressed (deduplicated by hash), and are garbage-collected on a configurable schedule.

Fine Tuned Weight

Mon, 01 Jan 0001 00:00:00 +0000

What is a Fine Tuned Weight?

A FineTunedWeight in OME is a Kubernetes resource that represents a set of weights that were fine-tuned from an existing Base Model. Rather than describing a full, standalone model, it points back to a base model and carries only the artifacts and metadata specific to the fine-tuning - for example a LoRA adapter, an added adapter module, or a distilled variant.

Serving Runtime

Tue, 14 Mar 2023 00:00:00 +0000

The only difference between the two is that one is namespace-scoped and the other is cluster-scoped.

A ClusterServingRuntime defines the templates for Pods that can serve one or more particular model. Each ClusterServingRuntime defines key information such as the container image of the runtime and a list of the models that the runtime supports. Other configuration settings for the runtime can be conveyed through environment variables in the container specification.

Inference Service

Tue, 14 Mar 2023 00:00:00 +0000

What is an InferenceService?

An InferenceService is the central Kubernetes resource in OME that orchestrates the complete lifecycle of model serving. It acts as a declarative specification that describes how you want your AI models deployed, scaled, and served across your cluster.

Think of InferenceService as the “deployment blueprint” for your AI workloads. It brings together models (defined by BaseModel/ClusterBaseModel), runtimes (defined by ServingRuntime/ClusterServingRuntime), and infrastructure configuration to create a complete serving solution.

Benchmark

Tue, 14 Mar 2023 00:00:00 +0000

A BenchmarkJob is a resource in OME that automates the performance benchmarking of inference service or OCI Generative AI Service endpoints. It allows you to evaluate model serving performance under various traffic patterns and load conditions.

BenchmarkJob uses genai-bench, a comprehensive benchmarking tool for evaluating generative AI model serving systems. For detailed information about genai-bench features and capabilities, refer to the official genai-bench documentation.

Core Components

A BenchmarkJob consists of several key components:

Ingress and External Access

Fri, 21 Jun 2024 00:00:00 +0000

What is Ingress in OME?

Ingress in OME provides external access to your AI inference services running inside the Kubernetes cluster. When you deploy an InferenceService, OME automatically creates the appropriate ingress resources based on your deployment mode and cluster configuration, allowing external clients to make API calls to your models.