Hopsworks Model Serving REST API#

Introduction#

Hopsworks provides model serving capabilities by leveraging KServe as the model serving platform and Istio as the ingress gateway to the model deployments.

This document explains how to interact with a model deployment via REST API.

Sending Inference Requests through Istio Ingress#

The full inference URL is constructed by combining a base path with a model server-specific suffix. See URL Paths for the complete URL format and examples.

Authentication#

All requests must include an API Key for authentication. You can create an API key by following this guide.

Include the key in the authorization header:

authorization: ApiKey <API_KEY_VALUE>

Headers#

URL Paths#

Deployed models are accessible through the Istio ingress gateway using path-based routing. The full URL is constructed by combining the base path with a model server-specific suffix. This URL is also provided on the model deployment page in the Hopsworks UI.

Where server-specific_suffix depends on the model server type (see ML Inference Paths or OpenAI-compatible Paths).

Base URL#

The base URL is composed of the Istio ingress gateway IP, the project name, and the deployment name.

Host: <deployment-name>.<project-name>.<knative-domain-name>

ML Inference#

For model deployments using Python, KServe sklearnserver, or TensorFlow Serving, the URL follows the KServe V1 inference protocol.

# Returns: https://<istio-host>/v1/<project>/<deployment>/v1/models/<name>:predict
inference_url = deployment.get_inference_url()

Deployment schema discovery#

Deployments that carry a deployment schema describe their request and response contract as JSON Schema and OpenAPI, and validate every request against it before any predictor code runs. The documents are served by the Hopsworks REST API (not the Istio ingress), with an API key that has the SERVING scope:

format is schema (default), jsonschema, or openapi; schemaId=<id> returns the contract of an earlier revision. The serving id is the id field of GET .../project/<project_id>/serving?name=<deployment_name>. A deployment without a schema, or an unknown id, answers 404 with error code 240037. Feature view deployments answer on the same <base_url>/v1/models/<name>:predict route as Python model deployments. See the Deployment Schema Guide.

OpenAI-compatible#

vLLM deployments provide an OpenAI API-compatible endpoint at <base_url>/v1/, allowing you to send any standard OpenAI API request to the vLLM server.

Refer to the official vLLM OpenAI-compatible server documentation for details about the available APIs.

# Returns: https://<istio-host>/v1/<project>/<deployment>/v1
# Append /chat/completions or /completions for specific endpoints
openai_url = deployment.get_openai_url()

Request Format#

The request format depends on the model server being used.

For predictive inference (TensorFlow, sklearn, or Python model server), the request must be sent as a JSON object containing an inputs or instances field. See more information on the request format.

import requests


data = {"inputs": [[4641025220953719, 4920355418495856]]}

headers = {"authorization": "ApiKey <your_api_key>", "content-type": "application/json"}

response = requests.post(
    "https://<ISTIO_GATEWAY_IP>/v1/my_project/fraud/v1/models/fraud:predict",
    headers=headers,
    json=data,
)
print(response.json())

curl -X POST "https://<ISTIO_GATEWAY_IP>/v1/my_project/fraud/v1/models/fraud:predict" \
  -H "authorization: ApiKey <your_api_key>" \
  -H "content-type: application/json" \
  -d '{
        "inputs": [
          [4641025220953719, 4920355418495856]
        ]
      }'

For generative inference (vLLM), the request follows the OpenAI specification supported by the vLLM OpenAI-compatible server.

import requests


data = {
    "model": "my-llm",
    "messages": [{"role": "user", "content": "Hello, how are you?"}],
}

headers = {"authorization": "ApiKey <your_api_key>", "content-type": "application/json"}

response = requests.post(
    "https://<ISTIO_GATEWAY_IP>/v1/my_project/my-llm/v1/chat/completions",
    headers=headers,
    json=data,
)
print(response.json())

curl -X POST "https://<ISTIO_GATEWAY_IP>/v1/my_project/my-llm/v1/chat/completions" \
-H "authorization: ApiKey <your_api_key>" \
-H "content-type: application/json" \
-d '{
        "model": "my-llm",
        "messages": [
            {"role": "user", "content": "Hello, how are you?"}
        ]
    }'

CORS#

The Istio EnvoyFilter handles CORS preflight (OPTIONS) requests automatically. Allowed origins can be configured via istio.envoyFilter.corsAllowedOrigins in the Helm chart configuration.

Response#

The model returns predictions in a JSON object. The response depends on the model server implementation.