Deploy a Simple Inference Service

Tue, 14 Mar 2023 00:00:00 +0000

This page shows you how to deploy a simple inference service using OME. You’ll learn how to create an InferenceService that serves a pre-trained model for real-time inference using SGLang and OpenAI-compatible APIs.

Before you begin

You need to have the following:

A Kubernetes cluster with OME installed
kubectl configured to communicate with your cluster
GPU nodes available in your cluster (A100, H100, H200, or B4)
Access to OME container registry (ghcr.io/sgl-project/)

Step 1: Verify prerequisites

Check that OME is installed and running:

Run Performance Benchmarks

Tue, 14 Mar 2023 00:00:00 +0000

This page shows you how to run performance benchmarks on your inference services using OME’s BenchmarkJob. You’ll learn how to test different traffic scenarios, measure performance metrics, and store results for analysis.

Before you begin

You need to have the following:

A Kubernetes cluster with OME installed
kubectl configured to communicate with your cluster
An InferenceService deployed and ready
Access to storage for benchmark results (OCI Object Storage or PVC)
OME benchmark tool image available

Step 1: Verify prerequisites

Check that your inference service is running:

Tasks on OME

Deploy a Simple Inference Service

Before you begin

Step 1: Verify prerequisites

Run Performance Benchmarks

Before you begin

Step 1: Verify prerequisites