How To Configure A Predictor#

Introduction#

In this guide, you will learn how to configure a predictor for a trained model.

Predictors are the main component of deployments. They are responsible for running a model server that loads a trained model, handles inference requests and returns predictions. They can be configured to use different model servers, serving tools, log specific inference data or scale differently. In each predictor, you can configure the following components:

GUI#

Step 1: Create new deployment#

If you have at least one model already trained and saved in the Model Registry, navigate to the deployments page by clicking on the Deployments tab on the navigation menu on the left.

Once in the deployments page, you can create a new deployment by either clicking on New deployment (if there are no existing deployments) or on Create new deployment it the top-right corner. Both options will open the deployment creation form.

Step 2: Choose a framework#

A simplified creation form will appear, including the most common deployment fields from all available configurations. The first step is to select your model to serve, which is done by first selecting the framework the model was registered as in the model registry.

For example if you registered the model as a TensorFlow model using ModelRegistry.tensorflow.create_model(...) you select Tensorflow in the dropdown.

All models registered for a specific framework will be listed in the model dropdown.

Moreover, you can optionally select a predictor script (see Step 3), enable KServe (see Step 4) or change other advanced configuration (see Step 5). Otherwise, click on Create new deployment to create the deployment for your model.

Step 3 (Optional): Select a predictor script#

For python models, if you want to use your own predictor script click on From project and navigate through the file system to find it, or click on Upload new file to upload a predictor script now.

Step 4 (Optional): Change predictor environment#

If you are using a predictor script it is also required to configure the inference environment for the predictor. This environment needs to have all the necessary dependencies installed to run your predictor script.

By default, we provide a set of environments like tensorflow-inference-pipeline, torch-inference-pipeline and pandas-inference-pipeline that serves this purpose for common machine learning frameworks.

To create your own it is recommended to clone the minimal-inference-pipeline and install additional dependencies for your use-case.

Step 5 (Optional): Enable KServe#

Other configuration such as the serving tool, is part of the advanced options of a deployment. To navigate to the advanced creation form, click on Advanced options.

Here, you change the serving tool for your deployment by enabling or disabling the KServe checkbox.

Step 6 (Optional): Other advanced options#

Additionally, you can adjust the default values of the rest of components:

Once you are done with the changes, click on Create new deployment at the bottom of the page to create the deployment for your model.

Code#

Step 1: Connect to Hopsworks#

import hopsworks

project = hopsworks.login()

# get Hopsworks Model Registry handle
mr = project.get_model_registry()

# get Hopsworks Model Serving handle
ms = project.get_model_serving()

Step 2 (Optional): Implement predictor script#

class Predict(object):

    def __init__(self):
        """ Initialization code goes here:
            - Download the model artifact
            - Load the model
        """
        pass

    def predict(self, inputs):
        """ Serve predictions using the trained model"""
        pass

Step 3 (Optional): Upload the script to your project#

uploaded_file_path = dataset_api.upload("my_predictor.py", "Resources", overwrite=True)
predictor_script_path = os.path.join("/Projects", project.name, uploaded_file_path)

Step 4: Define predictor#

my_model = mr.get_model("my_model", version=1)

my_predictor = ms.create_predictor(my_model,
                                   # optional
                                   model_server="PYTHON",
                                   serving_tool="KSERVE",
                                   script_file=predictor_script_path
                                   )

Step 5: Create a deployment with the predictor#

my_deployment = my_predictor.deploy()

# or
my_deployment = ms.create_deployment(my_predictor)
my_deployment.save()

API Reference#

Predictor

Model Server#

Hopsworks Model Serving currently supports deploying models with a Flask server for python-based models or TensorFlow Serving for TensorFlow / Keras models. Support for TorchServe for running PyTorch models is coming soon. Today, you can deploy PyTorch models as python-based models.

Serving tool#

In Hopsworks, model servers can be deployed in three different ways: directly on Docker, on Kubernetes deployments or using KServe inference services. Although the same models can be deployed in either of our two serving tools (Python or KServe), the use of KServe is highly recommended. The following is a comparative table showing the features supported by each of them.

Custom script#

Depending on the model server and serving tool used in the deployment, you can provide your own python script to load the model and make predictions.

Environment variables#

A number of different environment variables is available in the predictor to ease its implementation.

Transformer#

Transformers are used to apply transformations on the model inputs before sending them to the predictor for making predictions using the model. To learn more about transformers, see the Transformer Guide.

Inference logger#

Inference loggers are deployment components that log inference requests into a Kafka topic for later analysis. To learn about the different logging modes, see the Inference Logger Guide

Inference batcher#

Inference batcher are deployment component that apply batching to the incoming inference requests for a better throughput-latency trade-off. To learn about the different configuration available for the inference batcher, see the Inference Batcher Guide.

Resources#

Resources include the number of replicas for the deployment as well as the resources (i.e., memory, CPU, GPU) to be allocated per replica. To learn about the different combinations available, see the Resources Guide.

API protocol#

Hopsworks supports both REST and gRPC as the API protocols to send inference requests to model deployments. In general, you use gRPC when you need lower latency inference requests. To learn more about the REST and gRPC API protocols for model deployments, see the API Protocol Guide.

Conclusion#

In this guide you learned how to configure a predictor.