Python Deployment#

Introduction#

Python deployments allow you to deploy a Python script as a service without requiring a model artifact in the Model Registry. This is useful for custom inference pipelines, feature view deployments, or any Python-based program that needs to be served behind an HTTP endpoint.

In each Python deployment, you can configure the following:

Web UI#

A Python deployment and an agent deployment are the same thing: a Python script served as a long-running service without a model artifact. Whether the script is a plain HTTP server or an autonomous agent, it is created and managed the same way. In the current UI both are listed under Agent Deployments and created from the same form.

Step 1: Create a new deployment#

Navigate to Agent Deployments under the Agents section of the navigation menu on the left, then click on New agent.

Step 2: Configure the deployment#

Choose a name for your deployment. Under Source, keep Project file to run a script already in the project, or select Git repository to clone the script on every start. Then provide the script under Agent script file by clicking on From project, Upload new file or Create new file.

Step 3 (Optional): Change Python environment#

The script runs in one of the Python Environments available in your project. This environment must have all the necessary dependencies for your Python program.

Select an environment from the Python Environment dropdown. Hopsworks provides built-in environments such as python-agent-pipeline and the *-inference-pipeline environments, each with a different set of libraries pre-installed.

To create your own environment it is recommended to clone the minimal-inference-pipeline or pandas-inference-pipeline environment and install additional dependencies needed for your Python program.

Step 4 (Optional): Advanced configuration#

Click on advanced options to configure the deployment further, including:

Once you are done with the changes, click on Create at the bottom of the form to create the deployment.

Code#

Step 1: Connect to Hopsworks#

import hopsworks


project = hopsworks.login()

# get Hopsworks Model Serving handle
ms = project.get_model_serving()

Step 2: Implement a Python script#

import uvicorn
from fastapi import FastAPI


app = FastAPI()


@app.get("/ping")
async def ping():
    return {"status": "ready"}


@app.post("/echo")
async def echo(data: dict):
    return data


if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8080)

Step 3: Upload the script to your project#

import os

dataset_api = project.get_dataset_api()

uploaded_file_path = dataset_api.upload("python_server.py", "Resources", overwrite=True)
script_path = os.path.join("/Projects", project.name, uploaded_file_path)

Step 4: Create a deployment#

py_server = ms.create_endpoint(
    name="pyserver",
    script_file=script_path
)
py_deployment = py_server.deploy()

Step 5: Send requests#

import requests

url = py_deployment.get_endpoint_url()

response = requests.post(f"{url}/echo", json={"key": "value"})
print(response.json())

Environment variables#

A number of different environment variables is available in the Python deployment to ease its implementation.

Python environments#

Python deployments run in one of the *-inference-pipeline Python environments available in your project. Hopsworks provides built-in environments like minimal-inference-pipeline, pandas-inference-pipeline or torch-inference-pipeline with different sets of libraries pre-installed. By default, the pandas-inference-pipeline environment is used.

To create your own environment, it is recommended to clone the minimal-inference-pipeline or pandas-inference-pipeline environment and install additional dependencies needed for your Python program. To learn more about Python environments, see Python Environments.

Resources#

Configure CPU, memory, and GPU allocation for your Python deployment. Each deployment component has separate request and limit values.

For full details on resource configuration, see the Resources Guide.

Autoscaling#

Deployments use Knative Pod Autoscaler (KPA) to automatically scale the number of replicas based on traffic. You can configure the minimum and maximum number of instances as well as the scale metric (requests per second or concurrency).

For full details on autoscaling parameters, see the Autoscaling Guide.

Scheduling#

If the cluster has Kueue enabled, you can select a queue for your deployment from the advanced configuration. Queues control resource allocation and scheduling priority across the cluster.

For full details on scheduling configuration, see the Scheduling Guide.