Hopsworks Model Serving REST API#
Introduction#
Hopsworks provides model serving capabilities by leveraging KServe as the model serving platform and Istio as the ingress gateway to the model deployments.
This document explains how to interact with a model deployment via REST API.
Tutorials
End-to-end examples are available in the hopsworks-tutorials repository.
Sending Inference Requests through Istio Ingress#
The full inference URL is constructed by combining a base path with a model server-specific suffix. See URL Paths for the complete URL format and examples.
Authentication#
All requests must include an API Key for authentication. You can create an API key by following this guide.
Include the key in the authorization header:
Headers#
| Header | Description | Example Value |
|---|---|---|
authorization | API key for authentication. | ApiKey <your_api_key> |
content-type | Request payload type (JSON). | application/json |
Triton deployments also accept Triton's binary tensor extension on the Istio path, see Binary tensor data.
URL Paths#
Deployed models are accessible through the Istio ingress gateway using path-based routing. The full URL is constructed by combining the base path with a model server-specific suffix. This URL is also provided on the model deployment page in the Hopsworks UI.
<base_url>/<server-specific_suffix>
Where server-specific_suffix depends on the model server type (see ML Inference Paths, Open Inference Protocol (v2) Paths or OpenAI-compatible Paths).
Base URL#
The base URL is composed of the Istio ingress gateway IP, the project name, and the deployment name.
https://<ISTIO_GATEWAY_IP>/v1/<project_name>/<deployment_name>
Host-based routing (legacy)
Prior to path-based routing, requests were routed using a Host header matching the model deployment hostname, and https://<ISTIO_GATEWAY_IP> as base url.
Each model deployment gets its own Knative-generated hostname, and routing depends on the Host header matching Istio ingress gateway rules.
Path-based routing (described above) is the preferred method for external access.
ML Inference#
For model deployments using Python, KServe sklearnserver, or TensorFlow Serving, the URL follows the KServe V1 inference protocol.
Supported verbs and path format
| Model Server | Supported Verbs | Path Format |
|---|---|---|
| Python | predict | <base_url>/v1/models/<name>:<verb> |
| KServe sklearnserver | predict | <base_url>/v1/models/<name>:<verb> |
| TensorFlow Serving | predict, classify, regress | <base_url>/v1/models/<name>:<verb> |
Hopsworks Python API
ML inference urls can be retrieved using the Deployment class.
Open Inference Protocol (v2)#
Triton deployments follow the Open Inference Protocol, also called the KServe V2 protocol. The model name in the path is the deployment name.
Supported verbs and path format
| Model Server | Method | Path Format | Purpose |
|---|---|---|---|
| Triton | POST | <base_url>/v2/models/<name>/infer | Inference |
| Triton | GET | <base_url>/v2/models/<name>/ready | Model readiness |
| Triton | GET | <base_url>/v2/models/<name> | Model metadata |
The model metadata response lists the input and output tensors from config.pbtxt, with their names, data types and shapes.
Hopsworks Python API
The inference url of a Triton deployment is retrieved the same way.
Requests can also go through the Hopsworks REST API, which forwards them to the deployment, with the infer verb in place of predict:
POST https://<HOPSWORKS_HOST>/hopsworks-api/api/project/<project_id>/inference/serving/<deployment_name>:infer
The Hopsworks REST API accepts JSON bodies only, so binary tensor data works only on the Istio path.
Deployment schema discovery#
Deployments that carry a deployment schema describe their request and response contract as JSON Schema and OpenAPI, and validate every request against it before any predictor code runs. The documents are served by the Hopsworks REST API (not the Istio ingress), with an API key that has the SERVING scope:
GET https://<HOPSWORKS_HOST>/hopsworks-api/api/project/<project_id>/serving/<serving_id>/schema?format=openapi
format is schema (default), jsonschema, or openapi; schemaId=<id> returns the contract of an earlier revision. The serving id is the id field of GET .../project/<project_id>/serving?name=<deployment_name>. A deployment without a schema, or an unknown id, answers 404 with error code 240037. Feature view deployments answer on the same <base_url>/v1/models/<name>:predict route as Python model deployments. See the Deployment Schema Guide.
OpenAI-compatible#
vLLM deployments provide an OpenAI API-compatible endpoint at <base_url>/v1/, allowing you to send any standard OpenAI API request to the vLLM server.
e.g., Chat Completions endpoint
<base_url>/v1/chat/completions
Refer to the official vLLM OpenAI-compatible server documentation for details about the available APIs.
Hopsworks Python API
OpenAI-compatible urls can be retrieved using the Deployment class.
LLM Gateway#
On a cluster where the LLM Gateway is enabled, all the vLLM deployments of a project are also served by one OpenAI-compatible endpoint, <inference endpoint>/v1/<project>/v1/. The model of each request is the name of the deployment it goes to. Every deployment also stays on its own path.
The LLM Gateway also serves the Anthropic Messages API, with the base URL <inference endpoint>/v1/<project>/v1/anthropic that ms.get_anthropic_url() returns. A vLLM deployment serves it on its own path too, with the base URL <inference endpoint>/v1/<project>/<deployment>, verified from vLLM v0.28.0. Requests go to <base URL>/v1/messages, and Claude Code takes the base URL as ANTHROPIC_BASE_URL, see Coding agents.
Deployments the LLM Gateway does not serve
- A deployment in Knative mode.
- A deployment that scales to zero when idle (a minimum of 0 instances), or that autoscales on concurrency or requests per second.
- Every deployment of a project with a started deployment named
v1.
On a cluster with the LLM Gateway, a vLLM deployment created without a minimum number of instances gets a minimum of 1, so the LLM Gateway serves it.
Hopsworks Python API
The project's ModelServing gives the URL of the LLM Gateway, an OpenAI client for it, and its state.
ms = project.get_model_serving()
# The URL of the LLM Gateway, to use with any HTTP client
project_url = ms.get_openai_url()
# An OpenAI client for the LLM Gateway: choose the deployment by `model`
openai_client = ms.get_openai_client()
response = openai_client.chat.completions.create(
model="my_llm_deployment",
messages=[{"role": "user", "content": "Hello"}],
)
# The state of the LLM Gateway: enabled, available and url
state = ms.get_llm_gateway()
Deployment.get_openai_client() returns a client for one deployment's own path instead.
Turning the LLM Gateway off#
The LLM Gateway is on for every project and deployment by default, and a Data Owner can take deployments off it:
- For the whole project, with the LLM Gateway card in the project settings, or
ms.set_llm_gateway(False). The project's endpoint then answers404. - For one deployment, with the LLM Gateway switch, under Deployment configuration when creating the deployment and under Advanced configuration when editing it, or
deployment.set_llm_gateway(False).
No deployment restarts, every deployment keeps its own URL, and the change takes effect within a minute. A deployment's Overview shows whether the LLM Gateway serves it.
API reference
Request Format#
The request format depends on the model server being used.
For predictive inference (TensorFlow, sklearn, or Python model server), the request must be sent as a JSON object containing an inputs or instances field. See more information on the request format.
REST API example for Predictive Inference (Tensorflow or SkLearn or Python Serving)
import requests
data = {"inputs": [[4641025220953719, 4920355418495856]]}
headers = {"authorization": "ApiKey <your_api_key>", "content-type": "application/json"}
response = requests.post(
"https://<ISTIO_GATEWAY_IP>/v1/my_project/fraud/v1/models/fraud:predict",
headers=headers,
json=data,
)
print(response.json())
For Triton deployments, the request follows the Open Inference Protocol and is sent to the infer path. The body is a JSON object with an inputs list holding one tensor per model input named in config.pbtxt:
| Field | Description |
|---|---|
inputs[].name | Input name from config.pbtxt. |
inputs[].shape | Tensor shape, with the batch size first when the model has max_batch_size > 0. |
inputs[].datatype | Tensor data type, such as FP32, INT64 or BYTES. |
inputs[].data | Tensor values, flattened in row-major order. |
inputs[].parameters | Optional input parameters. |
outputs | Optional list of {"name": ...} objects selecting the outputs to return. |
parameters | Optional request parameters. |
REST API example for Triton
import requests
data = {
"inputs": [
{
"name": "input",
"shape": [1, 4],
"datatype": "FP32",
"data": [5.1, 3.5, 1.4, 0.2],
}
],
"outputs": [{"name": "label"}],
}
headers = {"authorization": "ApiKey <your_api_key>", "content-type": "application/json"}
response = requests.post(
"https://<ISTIO_GATEWAY_IP>/v1/my_project/irisonnx/v2/models/irisonnx/infer",
headers=headers,
json=data,
)
print(response.json())
curl -X POST "https://<ISTIO_GATEWAY_IP>/v1/my_project/irisonnx/v2/models/irisonnx/infer" \
-H "authorization: ApiKey <your_api_key>" \
-H "content-type: application/json" \
-d '{
"inputs": [
{"name": "input", "shape": [1, 4], "datatype": "FP32", "data": [5.1, 3.5, 1.4, 0.2]}
],
"outputs": [{"name": "label"}]
}'
curl -X POST "https://<HOPSWORKS_HOST>/hopsworks-api/api/project/<project_id>/inference/serving/irisonnx:infer" \
-H "authorization: ApiKey <your_api_key>" \
-H "content-type: application/json" \
-d '{
"inputs": [
{"name": "input", "shape": [1, 4], "datatype": "FP32", "data": [5.1, 3.5, 1.4, 0.2]}
]
}'
Triton answers with the v2 response:
{
"model_name": "irisonnx",
"model_version": "1",
"outputs": [
{"name": "label", "datatype": "INT64", "shape": [1], "data": [0]}
]
}
model_version is the Triton version folder that served the request, which is independent of the Hopsworks model version.
For generative inference (vLLM), the request follows the OpenAI specification supported by the vLLM OpenAI-compatible server.
vLLM chat completions
import requests
data = {
"model": "my-llm",
"messages": [{"role": "user", "content": "Hello, how are you?"}],
}
headers = {"authorization": "ApiKey <your_api_key>", "content-type": "application/json"}
response = requests.post(
"https://<ISTIO_GATEWAY_IP>/v1/my_project/my-llm/v1/chat/completions",
headers=headers,
json=data,
)
print(response.json())
Binary tensor data#
Triton's binary tensor data extension sends tensors as raw bytes after the JSON header, with the Inference-Header-Content-Length header giving the length of the JSON part. Send such requests to the Istio path only. The Hopsworks REST API takes the request body as JSON text, so it cannot forward binary tensor data.
CORS#
The Istio EnvoyFilter handles CORS preflight (OPTIONS) requests automatically. Allowed origins can be configured via istio.envoyFilter.corsAllowedOrigins in the Helm chart configuration.
Response#
The model returns predictions in a JSON object. The response depends on the model server implementation. Triton deployments return the v2 response shown in Request Format.