How To Configure A Predictor#

Introduction#

In this guide, you will learn how to configure a predictor for a trained model.

Predictors are the main component of deployments. They are responsible for running a model server that loads a trained model, handles inference requests and returns predictions. They can be configured to use different model servers, different resources or scale differently. In each predictor, you can decide the following configuration:

Web UI#

Step 1: Create new deployment#

If you have at least one model already trained and saved in the Model Registry, navigate to the deployments page by clicking on the Deployments tab on the navigation menu on the left.

Once in the deployments page, you can create a new deployment by either clicking on New deployment (if there are no existing deployments) or on Create new deployment it the top-right corner. Both options will open the deployment creation form.

Step 2: Choose a backend#

A simplified creation form will appear, including the most common deployment fields from all available configurations. The first step is to choose a backend for your model deployment. The backend will filter the models shown below according to the framework that the model was registered with in the model registry.

For example if you registered the model as a TensorFlow model using ModelRegistry.tensorflow.create_model(...) you select TensorFlow Serving in the dropdown.

All models compatible with the selected backend will be listed in the model dropdown.

After selecting a model from the dropdown, you can optionally choose a predictor script, modify the predictor environment, add a configuration file, or adjust other advanced settings as described in the optional steps below.

Otherwise, click on Create new deployment to create the deployment for your model with default values.

Step 3 (Optional): Select a predictor script#

For python models, to select a predictor script click on From project and navigate through the file system to find it, or click on Upload new file to upload a predictor script now.

Step 4 (Optional): Change predictor environment#

If you are using a predictor script, it is required to select an inference environment for the predictor. This environment needs to have all the necessary dependencies installed to run your predictor script.

Hopsworks provide a collection of built-in environments like minimal-inference-pipeline, pandas-inference-pipeline or torch-inference-pipeline with different sets of libraries pre-installed. By default, the pandas-inference-pipeline Python environment is used.

To create your own it is recommended to clone the pandas-inference-pipeline and install additional dependencies for your use-case.

Step 5 (Optional): Select a configuration file#

You can select a configuration file to be added to the artifact files. In Python model deployments, this configuration file will be available inside the model deployment at the local path stored in the CONFIG_FILE_PATH environment variable. In vLLM deployments, this configuration file will be directly passed to the vLLM server. You can find all configuration parameters supported by the vLLM server in the vLLM documentation.

Step 6 (Optional): Other advanced options#

To access the advanced deployment configuration, click on Advanced options.

Here, you can further change the default values of the predictor:

Once you are done with the changes, click on Create new deployment at the bottom of the page to create the deployment for your model.

Code#

Step 1: Connect to Hopsworks#

import hopsworks


project = hopsworks.login()

# get Hopsworks Model Registry handle
mr = project.get_model_registry()

# get Hopsworks Model Serving handle
ms = project.get_model_serving()

Step 2.1 (Optional): Implement a predictor script#

For Python model deployments, you need implement a predictor script that loads and serve your model.

class Predictor:
  def __init__(self):
      """Initialization code goes here"""
      # Optional __init__ params: project, deployment, model, async_logger
      # Model files can be found at os.environ["MODEL_FILES_PATH"]
      # self.model = ... # load your model

  def predict(self, inputs):
      """Serve predictions using the trained model"""
      # Use the model to make predictions
      # return self.model.predict(inputs)

class Predictor:
  def __init__(self, async_logger, model, project):
      """Initializes the serving state, reads a trained model"""
      # Get feature view attached to model
      ## self.model = model
      ## self.feature_view = model.get_feature_view()

      # Initialize feature view with async feature logger
      ## self.feature_view.init_feature_logger(feature_logger=async_logger)

  def predict(self, inputs):
      """Serves a prediction request usign a trained model"""
      # Extract serving keys and request parameters from inputs
      ## serving_keys = ...
      ## request_parameters = ...

      # Fetch feature vector with logging metadata
      ## vector = self.feature_view.get_feature_vector(serving_keys, 
      ##                                               request_parameters=request_parameters, 
      ##                                               logging_data=True)

      # Make predictions
      ## predictions = model.predict(vector)

      # Log Predictions
      ## self.feature_view.log(vector, 
      ##                       predictions=predictions, 
      ##                       model = self.model)

      # Predictions
      ## return predictions

class Predictor:
  def __init__(self):
      """Initialization code goes here"""
      # Optional __init__ params: project, deployment, model, async_logger
      # Model files can be found at os.environ["MODEL_FILES_PATH"]
      # self.model = ... # load your model

  async def predict(self, inputs):
      """Asynchronously serve predictions using the trained model"""
      # Perform async operations that required
      # result = await some_async_preprocessing(inputs)

      # Use the model to make predictions
      # return self.model.predict(result)

class Predictor:
    def __init__(self, project, model):
        # Access the project and model directly
        self.project = project
        self.model = model

Step 2.2 (Optional): Upload the script to your project#

uploaded_file_path = dataset_api.upload(
    "my_predictor.py", "Resources", overwrite=True
)
predictor_script_path = os.path.join(
    "/Projects", project.name, uploaded_file_path
)

Step 3: Pass predictor configuration to model deployment#

You can customize the default predictor settings when creating a model deployment.

my_model = mr.get_model("my_model", version=1)

my_deployment = my_model.deploy(
    # predictor configuration
    model_server="PYTHON",
    script_file=predictor_script_path,
)

API Reference#

Predictor

Model Server#

Hopsworks Model Serving supports deploying models with a Python model server for python-based models (scikit-learn, XGBoost , pytorch...), TensorFlow Serving for TensorFlow / Keras models and vLLM for Large Language Models (LLMs).

Each model server has specific requirements and supports different types of model artifacts, file formats, and configuration options. When deploying a model, ensure that your model files and configuration align with the expectations of the selected server.

All deployments use KServe as the serving platform, providing autoscaling (including scale-to-zero), fine-grained resource allocation, inference logging, inference batching, and transformers.

Predictor script#

For Python model deployments only, you can provide a custom Python script—called a predictor script—to load your model and serve predictions. This script is included in the artifact files of the deployment. The script must follow a specific template, as shown in Step 2.

Server configuration file#

For Python model deployments, you can provide a server configuration file to separate deployment-specific settings from the logic in your predictor or transformer scripts. This approach allows you to update configuration parameters without modifying the code. Within the deployment, the configuration file is accessible at the path specified by the CONFIG_FILE_PATH environment variable (see environment variables).

For vLLM deployments, the server configuration file is required and is used to configure the vLLM server. For example, you can use this configuration file to specify the chat template or LoRA modules to be loaded by the vLLM server. See all available parameters in the official documentation.

Environment variables#

A number of different environment variables is available in the predictor to ease its implementation.

Python environments#

Based on the model server used in the model deployment, you can select the Python environment where the predictor and transformer scripts will run. To create a new Python environment see Python Environments.

Transformer script#

Transformer scripts are Python scripts used to apply transformations on the model inputs before sending them to the predictor for making predictions using the model. To learn more about transformers, see the Transformer (KServe) Guide.

Inference logger#

Inference loggers are deployment components that log inference requests into a Kafka topic for later analysis.

To learn about the different logging modes, see the Inference Logger Guide

Inference batcher#

Inference batcher are deployment component that apply batching to the incoming inference requests for a better throughput-latency trade-off.

To learn about the different configuration available for the inference batcher, see the Inference Batcher Guide.

Resources#

Resources include the number of replicas for the deployment as well as the resources (i.e., memory, CPU, GPU) to be allocated per replica.

To learn about the different combinations available, see the Resources Guide.

Autoscaling#

Deployments use Knative Pod Autoscaler (KPA) to automatically scale the number of replicas based on traffic, including scale-to-zero.

To learn about the different autoscaling parameters, see the Autoscaling Guide.

Scheduling#

If the cluster has Kueue enabled, you can select a queue for your deployment from the advanced configuration. Queues control resource allocation and scheduling priority across the cluster.

For full details on scheduling configuration, see the Scheduling Guide.

API protocol#

Depending on the model server, Hopsworks supports both REST and gRPC as the API protocols to send inference requests to model deployments. In general, you use gRPC when you need lower latency inference requests.

To learn more about the REST and gRPC API protocols for model deployments, see the API Protocol Guide.