How To Export a Triton Model#
Introduction#
In this guide you will learn how to package a model for NVIDIA Triton Inference Server, register it in the Model Registry, deploy it and send it inference requests.
Triton is enabled per cluster
Triton deployments are available only on clusters where the administrator has enabled Triton and configured its image, see Triton Inference Server. On other clusters ms.get_triton_image_tags() returns an empty list, the UI does not offer the Triton model server, and the API rejects Triton deployments.
Python client 5.3 or later
Triton models need the Hopsworks Python client 5.3 or later, in the jobs and notebooks that read the model registry too. An older client raises ValueError: framework TRITON is not a supported framework whenever it reads a Triton model, for example when it lists the models of a registry that holds one.
Model directory#
A Triton model in Hopsworks is one Triton model directory per model version. save() uploads the contents of the directory you pass to it, so config.pbtxt sits at the top of that directory:
config.pbtxtis the Triton model configuration.- Each numeric folder (
1/,2/, ...) is a Triton model version and holds the model file of the backend, such asmodel.onnx,model.pt,model.planormodel.py. - A Python backend model can add a packed environment next to
config.pbtxt, see Python backend.
Hopsworks rejects a Triton deployment whose model files have no config.pbtxt or no numeric version folder. In the predictor, the files are placed at /mnt/models/<deployment-name>/, and Triton serves the model under the name of the deployment.
A minimal config.pbtxt for an ONNX model that takes four features and returns a label:
backend: "onnxruntime"
max_batch_size: 0
input [
{
name: "input"
data_type: TYPE_FP32
dims: [ -1, 4 ]
}
]
output [
{
name: "label"
data_type: TYPE_INT64
dims: [ -1 ]
}
]
Model name#
Omit name in config.pbtxt. Triton then takes the model name from the directory, which is the deployment name, so the same model version can be deployed under any deployment name. A name that differs from the deployment name makes Triton fail to load the model, and the deployment fails to start.
Versions#
The Hopsworks model version and the numeric Triton version folders are independent of each other. Triton serves the highest numeric folder only, unless version_policy in config.pbtxt selects other versions:
Hopsworks starts Triton with --model-control-mode=none. Triton loads the model at startup, and the model repository load and unload requests return errors. To serve a different model, register a new model version and deploy it.
Dynamic batching#
Triton batches requests in its own scheduler, configured in config.pbtxt. The KServe inference batcher is not available for Triton deployments, because Triton's scheduler already does this inside the model server.
With max_batch_size above 0, dims leave out the batch dimension, and the first entry of shape in a request is the batch size.
GPUs#
Request GPUs in the deployment's resources; Hopsworks gives Triton deployments no GPU by default. Which backends run on a GPU depends on the Triton image the administrator configured. Use instance_group in config.pbtxt to place model instances:
Python backend#
A Python backend model keeps its code in model.py inside the version folder. Dependencies that the Triton image does not contain ship as a conda-pack archive in the model directory:
Point EXECUTION_ENV_PATH in config.pbtxt at the archive:
backend: "python"
max_batch_size: 0
input [
{
name: "text"
data_type: TYPE_STRING
dims: [ -1 ]
}
]
output [
{
name: "scores"
data_type: TYPE_FP32
dims: [ -1, 2 ]
}
]
parameters: {
key: "EXECUTION_ENV_PATH"
value: { string_value: "$$TRITON_MODEL_DIRECTORY/environment.tar.gz" }
}
Pack the environment with the same Python version as the Python backend of the Triton image, for example Python 3.12 for the NVIDIA 26.x images. The backend runs model.py with the interpreter of the archive, and a model whose archive has another Python version fails to load.
conda create -y -n triton-env python=3.12
conda run -n triton-env pip install numpy torch transformers
conda install -y -n base -c conda-forge conda-pack
conda pack -n triton-env -o text_classifier/environment.tar.gz
A model.py implements TritonPythonModel:
import numpy as np
import triton_python_backend_utils as pb_utils
class TritonPythonModel:
def initialize(self, args):
# args["model_repository"] and args["model_version"] locate the files
self.model = ...
def execute(self, requests):
responses = []
for request in requests:
texts = pb_utils.get_input_tensor_by_name(request, "text").as_numpy()
scores = self.model([t.decode("utf-8") for t in texts])
output = pb_utils.Tensor("scores", np.asarray(scores, dtype=np.float32))
responses.append(pb_utils.InferenceResponse(output_tensors=[output]))
return responses
The Python backend hands input tensors to model.py on the CPU by default, so move them to the GPU in model.py when the model runs there. Hopsworks mounts a memory-backed /dev/shm in Triton predictors, which the Python backend uses to pass tensors between Triton and the model process. It counts against the memory of the deployment, so size the memory request for the tensors of the largest batch as well as the model.
Code#
Step 1: Connect to Hopsworks#
Step 2: Register the model directory#
Use the ModelRegistry.triton.create_model(..) function to register a model as a Triton model, then invoke save() with the path to the local Triton model directory.
Step 3: Deploy the model#
The image version is the tag of the Triton image the deployment runs, see Triton image version. ms.get_triton_image_tags() lists the tags the cluster advertises, highest first. Without triton_image_tag, the deployment gets the highest one.
from hsml.resources import PredictorResources, Resources
print(ms.get_triton_image_tags())
triton_deployment = triton_model.deploy(
name="irisonnx",
triton_image_tag="26.09-py3",
resources=PredictorResources(
num_instances=1,
requests=Resources(cores=1, memory=2048, gpus=0),
limits=Resources(cores=2, memory=4096, gpus=0),
),
)
triton_deployment.start(await_running=600)
The deployment name is also the Triton model name, so it follows the deployment name rules: letters and digits only.
Knative mode and large images
Triton images are large: NVIDIA's 26.09-py3 image is 8.4 GiB compressed for amd64. In Knative mode a deployment that scales up from zero on a node that has not pulled the image yet waits for the pull before it answers.
Step 4: Send an inference request#
Triton deployments use the Open Inference Protocol (KServe v2), over REST or gRPC. Pass one tensor per input named in config.pbtxt, with its shape, datatype and the values flattened in row-major order in data.
The response is the v2 response of Triton:
{
"model_name": "irisonnx",
"model_version": "1",
"outputs": [
{"name": "label", "datatype": "INT64", "shape": [1], "data": [0]}
]
}
See the REST API guide for the inference URL, the request through the Hopsworks REST API, and curl examples.
API reference
Supported features#
| Feature | Triton deployments | Why |
|---|---|---|
| REST | Open Inference Protocol v2, JSON | Triton's HTTP endpoint |
| gRPC | Open Inference Protocol v2 | Triton's gRPC endpoint, see API protocol |
| Binary tensor extension | Direct requests to the Istio gateway only | The Hopsworks REST API forwards JSON bodies only |
| CPU and GPU | Yes, no GPU by default | The backends that run on a GPU depend on the image |
| Standard and Knative modes | Yes | See Deployment mode |
| Dynamic batching | dynamic_batching in config.pbtxt | Triton batches in its own scheduler |
| KServe inference batcher | No | Triton batches in its own scheduler |
| Predictor script | No | The model directory holds the model code, for example model.py |
| Transformer | No | Over REST the transformer forwards requests in the KServe v1 format, which Triton does not serve; pre-process in the model directory, for example in model.py |
| Inference logging and feature logging | No | The logger parses JSON only, and Triton requests can carry binary tensors or use gRPC |
| Disk logging | No | The upload runs the Hopsworks library inside the serving container, which the Triton image does not have |
| Several models or ensembles per deployment | No | A deployment serves one Triton model, named after the deployment |
| Model load and unload requests | No | Triton runs with --model-control-mode=none |