Bases: ABC
Scaling configuration for a predictor or transformer.
property writable #
Whether every instance uploads its logs to the project's Logs dataset when it stops.
'ALL_REPLICAS' or 'NONE'. The backend rejects 'ALL_REPLICAS' for TensorFlow Serving and vLLM, and for a KServe Python deployment with no predictor script, because those runtime images do not ship the hopsworks SDK the upload runs. Unset means the backend default: on where it is supported, off everywhere else.
property writable #
Maximum number of instances to scale to.
Maximum allowed is configured in the cluster settings by the cluster administrator. Must be at least 1 and greater than or equal to min_instances. Defaults to the cluster maximum, except for LLM deployments in KServe Standard mode, which default to min_instances (fixed replica count).
property writable #
min_instances: int
Minimum number of instances to scale to.
KServe Knative deployments scale to zero when this is 0, and the cluster may require it. KServe Standard deployments do not scale to zero and need at least 1. Defaults to 0 for KServe Knative deployments when the cluster requires scale-to-zero, otherwise to 1.
property writable #
The percentage of the scale metric threshold that, when exceeded during the panic window, will trigger a scale-up event. Min is 1. Max is 200. Default is 200.
property writable #
The percentage of the stable window to use as the panic window during high load situations. Min is 1. Max is 100. Default is 10.
property writable #
The metric to use for scaling.
CONCURRENCY and RPS are Knative-only metrics for KServe Knative deployments. CPU and MEMORY drive CPU/memory-based autoscaling in KServe Standard mode. Standard deployments default to CPU when min_instances < max_instances; with min_instances == max_instances no autoscaler is configured and the metric is cleared.
property writable #
The amount of time in seconds the last instance must be kept before being scaled down to zero. Default is 0.
property writable #
The interval in seconds over which to calculate the average metric. Larger values result in smoother scaling but slower reaction times. Min is 1 second. Max is 3600 seconds.
property writable #
Target value for the selected scaling metric that the autoscaler should try to maintain.
For RPS, this is requests per second. For CONCURRENCY, this is the number of concurrent requests. For CPU and MEMORY, this is the utilization percentage.
staticmethod #
get_default_scaling_configuration(
serving_tool: str,
min_instances: int | None,
component_type: str = "predictor",
effective_knative_mode: bool = True,
enforce_scale_to_zero: bool = True,
) -> ComponentScalingConfig
Get the default scaling configuration based on the serving tool and number of instances.
| PARAMETER | DESCRIPTION |
|---|---|
serving_tool | the serving tool to use (e.g. kserve) TYPE: |
min_instances | minimum number of instances, or None to use the default TYPE: |
component_type | the component type (predictor or transformer) TYPE: |
effective_knative_mode | whether the deployment runs in KServe Knative mode. Only meaningful when TYPE: |
enforce_scale_to_zero | whether to reject a non-zero minimum when the cluster requires scale-to-zero for Knative deployments. Transformers are built before the deployment mode is known and skip this check. The backend validates the assembled deployment mode-aware. TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
ComponentScalingConfig | The default scaling configuration for the given serving tool. |
Bases: Enum
Whether a component archives its logs to the project's Logs dataset when an instance stops.
Each instance keeps its own logs on local disk and uploads them through the REST API when it stops, including stops the platform initiates such as scale-to-zero. Only deployments whose serving container runs a Hopsworks inference pipeline image support archiving, since the upload runs the hopsworks SDK from that image. The backend rejects ALL_REPLICAS for TensorFlow Serving and vLLM, and for a KServe Python deployment with no predictor script, which runs the sklearnserver runtime.
Bases: Enum
Scaling metric for a predictor or transformer.
CONCURRENCY and RPS are Knative-only metrics, valid for KServe Knative deployments. CPU and MEMORY drive CPU/memory-based autoscaling, valid for KServe Standard and non-KServe deployments. In KServe Standard mode a deployment with min_instances == max_instances runs a fixed replica count and ignores the metric.