Bases: Enum
Which autoscaler runs the scale metric of a KServe Standard-mode component.
KEDA scales on CPU, MEMORY, or the vLLM engine metrics (QUEUE_DEPTH, KV_CACHE_USAGE, RUNNING_REQUESTS, QUEUE_TIME, TIME_TO_FIRST_TOKEN, REQUEST_LATENCY) through a KEDA ScaledObject, and is the autoscaler wherever KEDA is installed (the default). HPA is KServe's own HorizontalPodAutoscaler on CPU or MEMORY, the fallback of a cluster without KEDA. Knative deployments always use the Knative autoscaler and reject this setting.
Bases: ABC
Scaling configuration for a predictor or transformer.
property writable #
KEDA only: further metrics scaled on next to scale_metric, as {"scale_metric", "target"} dicts.
KEDA sizes the deployment by whichever metric asks for the most instances, e.g. queue depth or KV-cache usage, whichever fires first. Metrics must be distinct; engine metrics need a vLLM predictor.
property writable #
autoscaler: Autoscaler | None
Which autoscaler runs the scale metric of a KServe Standard-mode component.
HPA is KServe's own HorizontalPodAutoscaler, KEDA a KEDA ScaledObject (needs KEDA installed in the cluster). CPU and MEMORY run under either; the vLLM engine metrics and the request metrics only under KEDA. Unset means the backend default: KEDA wherever it is installed, HPA otherwise. Cleared with the metric when min_instances == max_instances, and rejected for Knative deployments.
property writable #
cold_start_timeout_seconds: int | None
With a minimum of 0 instances: seconds (1-3600) a request is held while the deployment wakes.
Unset means 600. A request held longer fails; set it above the time the model takes to load.
property writable #
idle_cooldown_seconds: int | None
With a minimum of 0 instances: seconds (0-3600) without a request before the last instance is removed.
Unset means 300.
property writable #
idle_scale_to_zero: bool | None
Standard mode only: whether the deployment scales to zero when idle (min_instances=0).
Setting it to True sets the minimum to 0; the backend reports True whenever the minimum is 0. The KEDA HTTP add-on holds the first request until an instance is ready, so the deployment stays reachable at zero. Needs KEDA installed and the KEDA autoscaler, and a predictor without a transformer. Knative mode scales to zero on its own with min_instances=0. An LLM deployment reloads its model on every wake: the first request after an idle period takes as long as a cold start.
property writable #
Whether every instance uploads its logs to the project's Logs dataset when it stops.
'ALL_REPLICAS' or 'NONE'. The backend rejects 'ALL_REPLICAS' for TensorFlow Serving and vLLM, and for a KServe Python deployment with no predictor script, because those runtime images do not ship the hopsworks SDK the upload runs. Unset means the backend default: on where it is supported, off everywhere else.
property writable #
Maximum number of instances to scale to.
Maximum allowed is configured in the cluster settings by the cluster administrator. Must be at least 1 and greater than or equal to min_instances. Defaults to the cluster maximum, except for LLM deployments in KServe Standard mode, which default to min_instances (fixed replica count).
property writable #
min_instances: int
Minimum number of instances to scale to.
0 means scale to zero when idle: KServe Knative deployments through their own autoscaler (the cluster may require it), KServe Standard deployments through the KEDA HTTP add-on, which holds the first request until an instance is ready and runs one instance while active. In Standard mode it needs KEDA installed and a predictor without a transformer. Through the SDK the minimum defaults to 1 (resources.num_instances), so scale to zero is opt-in: set it to 0. The UI defaults a new deployment to 0 where KEDA is installed.
property writable #
The percentage of the scale metric threshold that, when exceeded during the panic window, will trigger a scale-up event. Min is 1. Max is 200. Default is 200.
property writable #
The percentage of the stable window to use as the panic window during high load situations. Min is 1. Max is 100. Default is 10.
property writable #
scale_down_stabilization_window_seconds: int | None
KEDA only: seconds (0-3600) the metric must stay below target before instances are removed.
Unset means the cluster default (300 s). GPU-bound LLM replicas are slow to bring back, so a long window avoids churn. Rejected with KServe's HPA, which ignores it.
property writable #
The metric to use for scaling.
CONCURRENCY and RPS are Knative's request metrics, available on every KServe Knative deployment. In KServe Standard mode the KEDA HTTP add-on measures them at the gateway, for the predictor of a deployment without a transformer on any model server but vLLM; elsewhere the backend rejects them. CPU and MEMORY drive CPU/memory-based autoscaling in KServe Standard mode, under KServe's HPA or KEDA. The vLLM engine metrics (QUEUE_DEPTH, KV_CACHE_USAGE, RUNNING_REQUESTS, QUEUE_TIME, TIME_TO_FIRST_TOKEN, REQUEST_LATENCY) are for LLM deployments in KServe Standard mode, scaled by KEDA. Standard deployments default to CPU when min_instances < max_instances (to QUEUE_DEPTH for a vLLM predictor on a cluster with KEDA); with min_instances == max_instances no autoscaler is configured and the metric is cleared.
property writable #
The amount of time in seconds the last instance must be kept before being scaled down to zero. Default is 0.
property writable #
scale_up_stabilization_window_seconds: int | None
KEDA only: seconds (0-3600) the metric must stay above target before instances are added.
Unset means the cluster default (0 s, react at once). Rejected with KServe's HPA, which ignores it.
property writable #
The interval in seconds over which to calculate the average metric. Larger values result in smoother scaling but slower reaction times. Min is 1 second. Max is 3600 seconds.
property writable #
Target value for the selected scaling metric that the autoscaler should try to maintain.
For RPS, this is requests per second per instance (default 200), in Knative mode or, through the KEDA HTTP add-on, for an eligible Standard-mode predictor. For CONCURRENCY, this is the number of concurrent requests per instance (default 100), under the same rule. For CPU and MEMORY, this is the utilization percentage. For QUEUE_DEPTH, this is the number of requests waiting in the vLLM engine queue per replica (default 5). For KV_CACHE_USAGE, this is the KV-cache utilization percentage per replica (default 80). For RUNNING_REQUESTS, this is the number of requests being generated per replica (default 32). For QUEUE_TIME, TIME_TO_FIRST_TOKEN and REQUEST_LATENCY, this is the average in milliseconds the autoscaler keeps the deployment under (defaults 1000, 2000 and 10000).
staticmethod #
get_default_scaling_configuration(
serving_tool: str,
min_instances: int | None,
component_type: str = "predictor",
effective_knative_mode: bool = True,
enforce_scale_to_zero: bool = True,
) -> ComponentScalingConfig
Get the default scaling configuration based on the serving tool and number of instances.
| PARAMETER | DESCRIPTION |
|---|---|
serving_tool | the serving tool to use (e.g. kserve) TYPE: |
min_instances | minimum number of instances, or None to use the default TYPE: |
component_type | the component type (predictor or transformer) TYPE: |
effective_knative_mode | whether the deployment runs in KServe Knative mode. Only meaningful when TYPE: |
enforce_scale_to_zero | whether to reject a non-zero minimum when the cluster requires scale-to-zero for Knative deployments. Transformers are built before the deployment mode is known and skip this check. The backend validates the assembled deployment mode-aware. TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
ComponentScalingConfig | The default scaling configuration for the given serving tool. |
Bases: Enum
Whether a component archives its logs to the project's Logs dataset when an instance stops.
Each instance keeps its own logs on local disk and uploads them through the REST API when it stops, including stops the platform initiates such as scale-to-zero. Only deployments whose serving container runs a Hopsworks inference pipeline image support archiving, since the upload runs the hopsworks SDK from that image. The backend rejects ALL_REPLICAS for TensorFlow Serving and vLLM, and for a KServe Python deployment with no predictor script, which runs the sklearnserver runtime.
Bases: Enum
Scaling metric for a predictor or transformer.
CONCURRENCY and RPS are Knative's metrics; in KServe Standard mode the KEDA HTTP add-on measures the same two at the gateway for a predictor without a transformer on any model server but vLLM (needs KEDA). CPU and MEMORY drive CPU/memory-based autoscaling, valid for KServe Standard and non-KServe deployments, under either autoscaler (see Autoscaler). The vLLM engine metrics apply to LLM deployments in KServe Standard mode and are read from Prometheus by KEDA, so they need the KEDA autoscaler and a cluster with KEDA installed: QUEUE_DEPTH (requests waiting in the engine queue per replica), RUNNING_REQUESTS (requests being generated per replica), KV_CACHE_USAGE (KV-cache utilization in percent), QUEUE_TIME (average time a request waits before generation starts, in ms), TIME_TO_FIRST_TOKEN (average time to the first token, in ms) and REQUEST_LATENCY (average end-to-end latency, in ms). The latency metrics are one-minute averages and scale the deployment in proportion to how far the average sits from its target. In KServe Standard mode a deployment with min_instances == max_instances runs a fixed replica count and ignores the metric.