hsml.model_serving #
ModelServing #
project_id property #
Id of the project in which Model Serving is located.
project_name property #
Name of the project in which Model Serving is located.
project_namespace property #
project_namespace: str | None
Kubernetes namespace of the project, or None if it was not provided.
project_path property #
Path of the project the registry is connected to.
create_deployment #
create_deployment(
predictor: Predictor,
name: str | None = None,
environment: str | None = None,
tags: tag.Tag
| dict[str, Any]
| list[tag.Tag | dict[str, Any]]
| None = None,
) -> Deployment
Create a Deployment metadata object.
Example
# login into Hopsworks using hopsworks.login()
# get Hopsworks Model Registry handle
mr = project.get_model_registry()
# retrieve the trained model you want to deploy
my_model = mr.get_model("my_model", version=1)
# get Hopsworks Model Serving handle
ms = project.get_model_serving()
my_predictor = ms.create_predictor(my_model)
my_deployment = ms.create_deployment(my_predictor)
my_deployment.save()
Using the model object
Using the Model Serving handle
# login into Hopsworks using hopsworks.login()
# get Hopsworks Model Registry handle
mr = project.get_model_registry()
# retrieve the trained model you want to deploy
my_model = mr.get_model("my_model", version=1)
# get Hopsworks Model Serving handle
ms = project.get_model_serving()
my_predictor = ms.create_predictor(my_model)
my_deployment = my_predictor.deploy()
my_deployment.get_state().describe()
Lazy
This method is lazy and does not persist any metadata or deploy any model. To create a deployment, call the save() method.
| PARAMETER | DESCRIPTION |
|---|---|
predictor | predictor to be used in the deployment TYPE: |
name | name of the deployment TYPE: |
environment | (Deprecated) The project Python environment to use. This argument will be ignored, use the argument TYPE: |
tags | Optionally the tags to attach to the deployment when it is created, in the same shapes accepted by feature groups. A single TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
Deployment | The deployment metadata object. |
create_endpoint #
create_endpoint(
name: str,
script_file: str,
description: str | None = None,
resources: PredictorResources | dict | None = None,
inference_logger: InferenceLogger
| dict
| str
| None = None,
inference_batcher: InferenceBatcher
| dict
| None = None,
api_protocol: str | None = IE.API_PROTOCOL_REST,
environment: str | None = None,
scaling_configuration: PredictorScalingConfig
| dict
| None = None,
env_vars: dict | None = None,
tracing: DeploymentTracingConfig | dict | None = None,
git_url: str | None = None,
git_provider: str | None = None,
git_branch: str | None = None,
git_auto_redeploy: bool = False,
knative_mode: bool | None = None,
memory: DeploymentMemoryConfig | dict | None = None,
) -> Predictor
Create an Entrypoint metadata object.
Example
Lazy
This method is lazy and does not persist any metadata or deploy any endpoint on its own. To create a deployment using this endpoint, call the deploy() method.
| PARAMETER | DESCRIPTION |
|---|---|
name | Name of the endpoint. TYPE: |
script_file | Path to a custom script file implementing a HTTP server, either local or already uploaded to HopsFS. TYPE: |
description | Description of the endpoint. TYPE: |
resources | Resources to be allocated for the predictor. TYPE: |
inference_logger | Inference logger configuration. TYPE: |
inference_batcher | Inference batcher configuration. TYPE: |
api_protocol | API protocol to be enabled in the deployment (i.e., 'REST' or 'GRPC'). TYPE: |
environment | The project Python environment to use TYPE: |
scaling_configuration | Scaling configuration for the predictor. TYPE: |
env_vars | Environment variables to set on the predictor. TYPE: |
tracing | Tracing configuration for the endpoint. TYPE: |
git_url | Optional Git repository URL for a git-backed endpoint. TYPE: |
git_provider | Git provider for git-backed endpoints. TYPE: |
git_branch | Optional branch to clone for git-backed endpoints. TYPE: |
git_auto_redeploy | Roll the endpoint to the branch HEAD whenever a new commit is pushed. Only valid together with TYPE: |
knative_mode | Whether to deploy in KServe Knative mode. TYPE: |
memory | Memory given to an agent that configures none itself; see TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
Predictor | The predictor metadata object. |
create_feature_view_predictor #
create_feature_view_predictor(
feature_view: Any,
name: str | None = None,
description: str | None = None,
training_dataset_version: int | None = None,
passed_features: list[str] | None = None,
schema: DeploymentSchema | dict | None = None,
script_file: str | None = None,
resources: PredictorResources | dict | None = None,
scaling_configuration: PredictorScalingConfig
| dict
| None = None,
environment: str | None = None,
env_vars: dict | None = None,
tags: tag.Tag
| dict[str, Any]
| list[tag.Tag | dict[str, Any]]
| None = None,
knative_mode: bool | None = None,
) -> Predictor
Create the predictor of a feature view deployment, which serves transformed feature vectors without a model.
Same contract as a model deployment with the default predictor; the response carries one feature vector per request row under predictions and the column names under columns.
Lazy
This method is lazy and does not persist any metadata or deploy anything on its own. To create a deployment using this predictor, call the deploy() method.
| PARAMETER | DESCRIPTION |
|---|---|
feature_view | The feature view to serve. TYPE: |
name | Deployment name; defaults to the view name and version without special characters. TYPE: |
description | Deployment description. TYPE: |
training_dataset_version | Training dataset whose statistics the transformations use; defaults to the last one accessed in this session. TYPE: |
passed_features | Features of the view whose values clients send with each request. |
schema | A refinement of the inferred deployment schema, keeping its fields. TYPE: |
script_file | A script subclassing TYPE: |
resources | Resources to be allocated for the predictor. TYPE: |
scaling_configuration | Scaling configuration for the predictor. TYPE: |
environment | The inference environment to use. TYPE: |
env_vars | Environment variables to set on the predictor. TYPE: |
tags | Tags to attach to the deployment when it is created. TYPE: |
knative_mode | Whether to deploy in KServe Knative mode; see TYPE: |
Example
| RETURNS | DESCRIPTION |
|---|---|
Predictor | The predictor metadata object. |
create_predictor #
create_predictor(
model: Model,
name: str | None = None,
artifact_version: str | None = None,
serving_tool: str | None = None,
script_file: str | None = None,
config_file: str | None = None,
resources: PredictorResources | dict | None = None,
inference_logger: InferenceLogger
| dict
| str
| None = None,
inference_batcher: InferenceBatcher
| dict
| None = None,
transformer: Transformer | dict | None = None,
api_protocol: str | None = IE.API_PROTOCOL_REST,
environment: str | None = None,
scaling_configuration: PredictorScalingConfig
| dict
| None = None,
env_vars: dict | None = None,
vllm_variant: str | None = None,
vllm_image_tag: str | None = None,
tracing: DeploymentTracingConfig | dict | None = None,
feature_logging: DeploymentLoggingConfig
| dict
| None = None,
tags: tag.Tag
| dict[str, Any]
| list[tag.Tag | dict[str, Any]]
| None = None,
schema: DeploymentSchema | dict | None = None,
passed_features: list[str] | None = None,
default_predictor: bool | None = None,
knative_mode: bool | None = None,
triton_image_tag: str | None = None,
) -> Predictor
Create a Predictor metadata object.
Example
# login into Hopsworks using hopsworks.login()
# get Hopsworks Model Registry handle
mr = project.get_model_registry()
# retrieve the trained model you want to deploy
my_model = mr.get_model("my_model", version=1)
# get Hopsworks Model Serving handle
ms = project.get_model_serving()
my_predictor = ms.create_predictor(my_model)
my_deployment = my_predictor.deploy()
Lazy
This method is lazy and does not persist any metadata or deploy any model on its own. To create a deployment using this predictor, call the deploy() method.
| PARAMETER | DESCRIPTION |
|---|---|
model | Model to be deployed. TYPE: |
name | Name of the predictor. TYPE: |
artifact_version | (Deprecated) Ignored. Deployments are versioned as a whole, see TYPE: |
serving_tool | Serving tool used to deploy the model server. TYPE: |
script_file | Path to a custom predictor script implementing the Predict class, either local or already uploaded to HopsFS. The script must implement a TYPE: |
config_file | Model server configuration file to be passed to the model deployment, either local or already uploaded to HopsFS. It can be accessed via TYPE: |
resources | Resources to be allocated for the predictor. TYPE: |
inference_logger | Inference logger configuration. TYPE: |
inference_batcher | Inference batcher configuration. TYPE: |
transformer | Transformer to be deployed together with the predictor. TYPE: |
api_protocol | API protocol of the deployment, 'REST' or 'GRPC'. Defaults to 'REST', which is the protocol TYPE: |
environment | The project Python environment to use TYPE: |
scaling_configuration | Scaling configuration for the predictor. TYPE: |
env_vars | Environment variables to set on the predictor. TYPE: |
vllm_variant | vLLM image variant for vLLM deployments. One of TYPE: |
vllm_image_tag | vLLM image tag override. If set, it must be one of the tags TYPE: |
tracing | Tracing configuration for the predictor. TYPE: |
feature_logging | Feature logging configuration for the predictor and its feature-log sidecar; see TYPE: |
tags | Optionally the tags to attach to the deployment when it is created, in the same shapes accepted by feature groups. A single TYPE: |
schema | Deployment schema describing the prediction requests; see TYPE: |
passed_features | Feature view features whose values clients send with each request; see |
default_predictor | Whether the library's default predictor serves the model; see TYPE: |
knative_mode | Whether to deploy in KServe Knative mode. TYPE: |
triton_image_tag | Triton Inference Server image tag for Triton deployments. If set, it must be one of the tags TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
Predictor | The predictor metadata object. |
create_transformer #
create_transformer(
script_file: str | None = None,
resources: PredictorResources | dict | None = None,
scaling_configuration: TransformerScalingConfig
| dict
| None = None,
env_vars: dict | None = None,
environment: str | None = None,
) -> Transformer
Create a Transformer metadata object.
Example
# login into Hopsworks using hopsworks.login()
# get Hopsworks Model Serving handle
ms = project.get_model_serving()
# create my_transformer.py Python script
class Transformer(object):
def __init__(self):
''' Initialization code goes here '''
pass
def preprocess(self, inputs):
''' Transform the requests inputs here. The object returned by this method will be used as model input to make predictions. '''
return inputs
def postprocess(self, outputs):
''' Transform the predictions computed by the model before returning a response '''
return outputs
# Local path — auto-uploaded on save.
my_transformer = ms.create_transformer(script_file="my_transformer.py")
# Or an already-uploaded HopsFS path:
my_transformer = ms.create_transformer(
script_file="/Projects/<project>/Resources/my_transformer.py"
)
# Or the Transformer class directly:
from hsml.transformer import Transformer
my_transformer = Transformer(script_file="my_transformer.py")
Create a deployment with the transformer
Lazy
This method is lazy and does not persist any metadata or deploy any transformer. To create a deployment using this transformer, set it in the predictor.transformer property.
| PARAMETER | DESCRIPTION |
|---|---|
script_file | Path to a custom predictor script implementing the Transformer class, either local or already uploaded to HopsFS. TYPE: |
resources | Resources to be allocated for the transformer. TYPE: |
scaling_configuration | Scaling configuration for the transformer. TYPE: |
env_vars | Environment variables to set on the transformer. TYPE: |
environment | The project Python environment the transformer runs in; defaults to the predictor's, or to the one the deployment names when the predictor runs a fixed runtime image and has none of its own. TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
Transformer | The transformer metadata object. |
deploy_agent #
deploy_agent(
entry: str,
name: str | None = None,
requirements: str | None = None,
environment: str | None = None,
upload_dir: str = "Resources/agents",
description: str | None = None,
resources: PredictorResources | dict | None = None,
inference_logger: InferenceLogger
| dict
| str
| None = None,
inference_batcher: InferenceBatcher
| dict
| None = None,
api_protocol: str | None = IE.API_PROTOCOL_REST,
scaling_configuration: PredictorScalingConfig
| dict
| None = None,
tracing: DeploymentTracingConfig | dict | None = None,
git_url: str | None = None,
git_provider: str | None = None,
git_branch: str | None = None,
git_auto_redeploy: bool = False,
knative_mode: bool | None = None,
memory: DeploymentMemoryConfig | dict | None = None,
framework: str | None = None,
chat_config: dict[str, Any] | str | None = None,
) -> Deployment
Deploy a Python script or package as an agent.
The agent is created on first call and updated on subsequent calls. Each call uploads the latest local code, refreshes the Python environment, and rewrites the deployment's predictor metadata to reflect the arguments passed in — including any unspecified arguments, which fall back to their defaults. The deployment's running state is left untouched; call start() after the first deploy and restart() to roll a running agent onto the new code. Works the same whether invoked from outside or inside a Hopsworks cluster.
Pass either a .py script or a directory containing a pyproject.toml. For a script, the file is uploaded and run directly. For a package, a wheel is built locally with the project's PEP 517 backend, uploaded, and installed; a small runner module invokes the package via runpy.run_module. When git_url is set, entry must be a safe relative .py path inside the repository and is used as-is.
ms = project.get_model_serving()
agent = ms.deploy_agent(entry="my_agent.py")
agent.start() # or agent.restart()
# iterate: edit code locally, push, then roll the running agent onto it
agent = ms.deploy_agent(entry="my_agent.py")
agent.restart()
| PARAMETER | DESCRIPTION |
|---|---|
entry | Local path to a TYPE: |
name | Name of the deployment, also used as the default Python environment name. Defaults to the basename of TYPE: |
requirements | Local path to a TYPE: |
environment | Name of the Python environment to use; defaults to TYPE: |
upload_dir | Directory in the Hopsworks Filesystem under which agent files are placed; the agent gets its own subdirectory TYPE: |
description | Description of the deployment. TYPE: |
resources | Resources to be allocated for the predictor. TYPE: |
inference_logger | Inference logger configuration. TYPE: |
inference_batcher | Inference batcher configuration. TYPE: |
api_protocol | API protocol to be enabled in the deployment (i.e., 'REST' or 'GRPC'). TYPE: |
scaling_configuration | Scaling configuration for the predictor. TYPE: |
tracing | Tracing configuration for the deployment. TYPE: |
git_url | Optional Git repository URL. When set, TYPE: |
git_provider | Git provider for git-backed agent deployments. TYPE: |
git_branch | Optional branch to clone for git-backed agent deployments. TYPE: |
git_auto_redeploy | Roll the agent to the branch HEAD whenever a new commit is pushed. Only valid together with TYPE: |
knative_mode | Whether to deploy in KServe Knative mode. TYPE: |
memory | Memory the deployment gives the agent, for an agent that configures none in its own code; see TYPE: |
framework | Framework the agent is built with: TYPE: |
chat_config | Chat contract of the agent, telling the chat panel and chat clients how to talk to it. A protocol name ( |
| RETURNS | DESCRIPTION |
|---|---|
Deployment | The deployment metadata object. |
| RAISES | DESCRIPTION |
|---|---|
ValueError | If |
hopsworks.client.exceptions.RestAPIError | If the backend encounters an error when handling the request. |
get_anthropic_url #
get_anthropic_url() -> str | None
Get the Anthropic-compatible base URL of the LLM gateway.
This is the value for ANTHROPIC_BASE_URL, used by Claude Code and the Anthropic SDK. The URL is the OpenAI-compatible URL of the LLM gateway followed by /anthropic. Requests are routed to a deployment by the model field of the request body, which is the deployment name. The same availability notes as ModelServing.get_openai_url apply.
| RETURNS | DESCRIPTION |
|---|---|
str | None | The Anthropic-compatible base URL, or |
| RAISES | DESCRIPTION |
|---|---|
hopsworks.client.exceptions.ModelServingException | If the Kubernetes namespace of the project is unknown. |
get_deployment #
get_deployment(name: str = None) -> Deployment | None
Get a deployment by name from Model Serving.
Example
Getting a deployment from Model Serving means getting its metadata handle so you can subsequently operate on it (e.g., start or stop).
| PARAMETER | DESCRIPTION |
|---|---|
name | Name of the deployment to get. TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
Deployment | None | The deployment metadata object or |
| RAISES | DESCRIPTION |
|---|---|
hopsworks.client.exceptions.RestAPIError | If unable to retrieve deployment from model serving. |
get_deployment_by_id #
get_deployment_by_id(id: int) -> Deployment | None
Get a deployment by id from Model Serving.
Getting a deployment from Model Serving means getting its metadata handle so you can subsequently operate on it (e.g., start or stop).
Example
| PARAMETER | DESCRIPTION |
|---|---|
id | Id of the deployment to get. TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
Deployment | None | The deployment metadata object or |
| RAISES | DESCRIPTION |
|---|---|
hopsworks.client.exceptions.RestAPIError | If unable to retrieve deployment from model serving. |
get_deployments #
get_deployments(
model: Model = None, status: str = None
) -> list[Deployment]
Get all deployments from model serving.
Example
# login into Hopsworks using hopsworks.login()
# get Hopsworks Model Registry handle
mr = project.get_model_registry()
# get Hopsworks Model Serving handle
ms = project.get_model_serving()
# retrieve the trained model you want to deploy
my_model = mr.get_model("my_model", version=1)
list_deployments = ms.get_deployment(my_model)
for deployment in list_deployments:
print(deployment.get_state())
| PARAMETER | DESCRIPTION |
|---|---|
model | Filter by model served in the deployments TYPE: |
status | Filter by status of the deployments TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
list[Deployment] | A list of deployments. |
| RAISES | DESCRIPTION |
|---|---|
hopsworks.client.exceptions.RestAPIError | If unable to retrieve deployments from model serving. |
get_inference_endpoints #
get_inference_endpoints() -> list[InferenceEndpoint]
Get all inference endpoints available in the current project.
| RETURNS | DESCRIPTION |
|---|---|
list[InferenceEndpoint] | Inference endpoints for model inference |
get_llm_gateway #
Get the state of the LLM gateway.
| RETURNS | DESCRIPTION |
|---|---|
dict[str, Any] | A dict with the keys |
| RAISES | DESCRIPTION |
|---|---|
hopsworks.client.exceptions.RestAPIError | In case the backend encounters an issue. |
get_openai_client #
get_openai_client(**kwargs: Any) -> openai.OpenAI
Get an OpenAI client for the LLM gateway, shared by the vLLM deployments of the project.
The client authenticates with the Hopsworks token of the current connection. Pass the deployment name as model in each request to choose the deployment. Requires the openai package, installed with pip install "hopsworks[openai]". Keyword arguments are passed to openai.OpenAI, for example timeout, max_retries or http_client. default_headers are merged over the authorization header set by Hopsworks. base_url and api_key are set by Hopsworks and cannot be passed. Without an http_client, certificates are verified as the main Hopsworks connection does. With hostname_verification=True, the certificate of the inference endpoint must cover its host; otherwise pass an http_client with a matching CA, or one with verify=False.
The endpoint answers 404 until it is available
The LLM gateway answers 404 until a vLLM deployment of the project is running and the cluster has the LLM gateway enabled. The per-deployment client from Deployment.get_openai_client keeps working in the meantime.
Only some vLLM deployments are served
The LLM gateway serves the Standard-mode vLLM deployments of the project whose LLM gateway switch is on and that do not scale to zero. Knative-mode deployments, deployments with a minimum of 0 instances and deployments with the switch off answer 404 here; use Deployment.get_openai_client for them.
| RETURNS | DESCRIPTION |
|---|---|
openai.OpenAI | The configured OpenAI client. |
| RAISES | DESCRIPTION |
|---|---|
hopsworks.client.exceptions.ModelServingException | If no endpoint URL is available, or the Kubernetes namespace of the project is unknown. |
ValueError | If |
ModuleNotFoundError | If the |
get_openai_url #
get_openai_url() -> str | None
Get the OpenAI-compatible URL of the LLM gateway, shared by the vLLM deployments of the project.
Requests are routed to a deployment by the model field of the request body, which is the deployment name. The URL stays the same when deployments are created, stopped or deleted.
The endpoint answers 404 until it is available
The LLM gateway answers 404 until a vLLM deployment of the project is running and the cluster has the LLM gateway enabled. The per-deployment URL from Deployment.get_openai_url keeps working in the meantime.
Only some vLLM deployments are served
The LLM gateway serves the Standard-mode vLLM deployments of the project whose LLM gateway switch is on and that do not scale to zero. Knative-mode deployments, deployments with a minimum of 0 instances and deployments with the switch off answer 404 here; use Deployment.get_openai_url for them.
| RETURNS | DESCRIPTION |
|---|---|
str | None | The OpenAI-compatible URL, or |
| RAISES | DESCRIPTION |
|---|---|
hopsworks.client.exceptions.ModelServingException | If the Kubernetes namespace of the project is unknown. |
get_triton_image_tags #
Get the Triton Inference Server image tags a cluster administrator advertises.
Use this to pick a valid triton_image_tag when creating a Triton deployment. Triton is opt-in per cluster and runs an image the administrator supplies, so an empty list means Triton is not enabled and creating a Triton deployment fails. A deployment keeps the tag it was created with even if that tag stops being advertised, so this list can be missing the tag an existing deployment uses.
Example
| RETURNS | DESCRIPTION |
|---|---|
list[str] | Advertised image tags. Empty if Triton is not enabled on this cluster. |
get_vllm_image_tags #
Get the vLLM runtime image tags a cluster administrator advertises.
Use this to pick a valid vllm_image_tag when creating an LLM deployment. Tags come back newest first, so the first one is what a deployment gets when no tag is given. A deployment keeps the tag it was created with even if that tag stops being advertised, so this list can be missing the tag an existing deployment uses.
Example
| PARAMETER | DESCRIPTION |
|---|---|
variant | The vLLM variant to list tags for, TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
list[str] | Advertised image tags, newest first. Empty if the variant advertises none. |
set_llm_gateway #
Switch the LLM gateway on or off for the whole project.
While it is off, the LLM gateway answers 404 for every deployment of the project. The per-deployment URLs keep working, and no deployment restarts. Requires the data owner role in the project. When the backend stores the switch but cannot apply it yet, it applies it within a minute and the call logs this at INFO level.
| PARAMETER | DESCRIPTION |
|---|---|
enabled | Whether the LLM gateway serves the deployments of the project. TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
dict[str, Any] | The state of the LLM gateway, as returned by |
| RAISES | DESCRIPTION |
|---|---|
hopsworks.client.exceptions.RestAPIError | In case the backend encounters an issue. |