Upgrading Hopsworks#
Introduction#
Upgrade a self-managed Hopsworks cluster one minor release at a time, in order. Skipping a minor release is not supported, and for OpenSearch it fails at startup. Upgrade from the latest patch release of your current minor release to the latest patch release of the next one. Patch releases carry fixes for the upgrade itself, and these notes are written for the latest ones: the move to OpenSearch 3.8, for example, needs 5.2.1 or later. This page covers upgrades from 5.0.x and later. The commands assume Hopsworks is installed in the hopsworks namespace.
| Upgrade | OpenSearch | What needs action |
|---|---|---|
| 5.0.x to 5.1.x | 1.3.x to 2.19.x | Kubernetes 1.34 or later before the upgrade, a first Helm 4 upgrade, and from 5.0.13 or earlier to 5.1.2 or later the Trino query history. |
| 5.1.x to 5.2.x | 2.19.x to 3.8.0: upgrade to 5.2.1 or later | The index pass, Kafka moves to KRaft, KEDA, Kubernetes 1.37 only afterwards, Trino, from 5.2.2 OnlineFS and the schema migrations, from 5.2.3 with your own image registry the Envoy image of Python apps, and from 5.2.5 Istio 1.30. |
| 5.2.x to 5.3.x | stays on 3.8.0 | Additional prerequisites, Strimzi 1.3 with central before its satellites, ICEBERG feature groups in Trino, the Filebeat registry, with your own pipelines, outputs or plugins Logstash 9, the memory of the OTLP sidecar of agent deployments, from 5.2.5 or later Istio 1.31, Envoy Gateway and LLM Gateway when turned on, with an external Kafka the topics Hopsworks creates and deletes on it, notification topics and shared feature stores, from 5.2.0 the subset shares in Trino and the OpenSearch security configuration, and from 5.2.0 or 5.2.1 OnlineFS and the schema migrations. |
Before every upgrade#
- Take a backup and check that it completed, as described in Disaster Recovery.
- Pass
helm upgrade --timeoutlong enough for the hooks of that upgrade. The upgrade from 5.1.x to 5.2.x has two that can run for hours: the OpenSearch index pass and the Kafka migration. - If you run satellite clusters, read the order in which they upgrade relative to the central cluster, for OpenSearch, for Kafka from 5.1.x to 5.2.x and for Strimzi 1.3, and leave KEDA out of them.
- If this is the first upgrade of the release with Helm 4, read Upgrading with Helm 4 first.
5.0.x to 5.1.x#
An upgrade from 5.0.13 or earlier to 5.1.2 or later also stops the Trino query history and strips the details of recorded queries after 7 days, which cannot be undone, as described in Trino query history.
Kubernetes 1.34#
5.1 requires Kubernetes 1.34 or later, up from 1.27 in 5.0, and helm upgrade refuses an older Kubernetes cluster. Upgrade Kubernetes first: 5.0.1 and later support up to 1.36, and 5.0.0 up to 1.34.
OpenSearch 2.19#
OpenSearch goes from 1.3.x to 2.19.x. The indices stay as they are, and the upgrade to 5.2 deals with those created by 1.x. Upgrade from 5.0 to 5.1.2 or later, which avoids the known issues of 5.1.0 and 5.1.1, as described in Upgrading OpenSearch (step 2) and Security configuration. Then confirm that OpenSearch 2.19 is live and green before the next upgrade, as step 2 there shows.
First upgrade with Helm 4#
The chart changes described in Upgrading with Helm 4 ship from 5.2.0, so a 5.1.x chart still declares .spec.replicas next to the HPA of an autoscaled workload. A server-side Helm 4 upgrade to 5.1.x therefore fails with conflict with "kube-controller-manager" with subresource "scale" on a workload its HPA has rescaled. A release installed by Helm 3 stays client-side under Helm 4 unless you pass --server-side=true, and does not meet that conflict. An upgrade from 5.0 to 5.1 also conflicted on the trino-password-file and trino-groups-file Secrets, whose fields the backend owns. Running it again with --force-conflicts overwrites the Trino user database that the backend wrote (password.db and group.db), so check the Trino users afterwards. --force-conflicts applies to the whole release, as The RSS ConfigMap explains.
Satellite clusters on 5.0#
Upgrade the central cluster first, as described in Satellite clusters. Until a satellite is on 5.1, the central cluster's 5.1 Python client writes to the satellite's 5.0 online store and searches its OpenSearch 1.3. The following restrictions apply during this window, also for workloads that use no 5.1 feature:
- Similarity search fails with
[knn] unknown token [START_OBJECT] after [filter]on an embedding feature group created after the central upgrade. The 5.1 central cluster creates its index on faiss, and the client puts the search filters inside the nearest-neighbor query for faiss, which OpenSearch 1.3 cannot parse. Embedding feature groups created before the central upgrade keep working, and search recovers once the satellite is on 5.1. - In a stream feature group, an insert with several rows for one primary key leaves the last of those rows in the online store instead of the one with the latest event time. The 5.1 client marks the older rows for the offline store only, in a way the 5.0 online store does not read. The offline store is not affected, and feature groups that are not stream feature groups drop those rows in the client. The rows stay as they are after the satellite's upgrade, so insert the newest values of the affected keys again once the satellite is on 5.1.
- Creating a new cosine embedding feature group fails while the satellite runs OpenSearch 1.3, for both shared project indexes and dedicated indexes. The central cluster creates the embedding fields with the faiss engine, whose OpenSearch 1.3 implementation does not support the
cosinesimilspace type. Postpone creation of these feature groups until the target satellite is upgraded to 5.1 and runs OpenSearch 2.19.
Deletes and upsert_if_newer work as on 5.1.
5.1.x to 5.2.x#
OpenSearch 3.8.0#
Upgrade a 5.1.x cluster to 5.2.1 or later. OpenSearch 3.8.0 refuses to start on an index created before OpenSearch 2.0, so a pre-upgrade hook copies those indices on the running 2.19 cluster first. Follow Upgrading OpenSearch: it covers what keeps working during the pass, the timeout, the report, vector ingestion and the steps.
Kafka 4 and KRaft#
The upgrade migrates every Kafka from ZooKeeper to KRaft and moves Strimzi from 0.45 to 1.2, or to 1.3 from the 5.2.x patch release that uses it. Run the read-only checks in Before you start first, and read Kafka 4 and KRaft upgrade notes, including the step that ArgoCD needs and, with Strimzi 1.2, its ignoreDifferences entry.
Migration V109 waits for the statistics cleaner#
Up to 5.1.2, a query of the statistics cleaner can take up to an hour on a cluster with many feature statistics; 5.1.3 fixes it. The 5.1.x pods keep running while the 5.2 schema migrations run, so each statement of migration V109 waits for that query on an upgrade from 5.1.2 or earlier to 5.2.0 or 5.2.1, or to any 5.2 release on a cluster that disables the DDL SQL node. From 5.2.2 the migrations run on the DDL SQL node and do not wait for it, as described in Schema migrations run on the DDL SQL node. When you upgrade to 5.2.0 or 5.2.1, or with the DDL SQL node disabled, upgrade to the latest 5.1.x patch first.
If V109 is already waiting, Waiting for table metadata lock shows in the processlist next to a SELECT fds.* FROM hopsworks.feature_descriptive_statistics. Scale hopsworks-instance to 0, KILL that query on its SQL node, and scale hopsworks-instance back up once Flyway has finished.
KEDA#
From 5.2 the chart installs KEDA 2.21 and its HTTP add-on by default (hopsworks.variables.kube_keda_installed: true), so this upgrade adds them to every existing release. KEDA Autoscaling describes what the chart installs and what KEDA needs from the cluster.
- One external metrics provider per Kubernetes cluster. A Kubernetes cluster that already runs KEDA, or the OpenShift Custom Metrics Autoscaler, must have it removed first, or leave the subchart off with
hopsworks.variables.kube_keda_installed: false, as described in Cluster requirements. - The CRDs arrive through a pre-upgrade hook. The
keda-crd-upgradehook creates the KEDA CRDs before the operators start, as described in CustomResourceDefinitions. It needs the toolbox image, from 5.2.2 also thehopsworks/keda-crdsimage it reads the CRDs from, and a few seconds of the--timeout, and no step beforehelm upgrade. A failed upgrade to 5.2.0 or 5.2.1 can leave a ConfigMap behind, which CustomResourceDefinitions says how to delete. - Six more images to mirror on an airgapped cluster, seven from 5.2.2:
ghcr.io/kedacore/keda,keda-metrics-apiserverandkeda-admission-webhooks,ghcr.io/kedacore/http-add-on-operator,http-add-on-interceptorandhttp-add-on-scaler, and from 5.2.2hopsworks/keda-crds. What the chart installs names the values that override them. - The upgrade does not change deployments. A Standard-mode deployment keeps its stored scaling configuration. Where it named no autoscaler, KEDA runs it from its next start, with the same CPU or memory threshold as before. Nothing scales to zero until a minimum of 0 is set.
- Uninstall takes longer, because a cleanup hook waits for the KEDA operators before the release is deleted, as described in Uninstalling.
Kubernetes 1.37#
5.2 supports Kubernetes up to 1.37, but 5.1.x stops at 1.36. Upgrade Hopsworks to 5.2.x before you move the Kubernetes cluster to 1.37.
Under ArgoCD before 3.5.0, an Application with the ServerSideApply=true sync option stops syncing on Kubernetes 1.37. Kubernetes 1.37 adds spec.preventPodSchedulingIfMissing to every CSIDriver, and the schema built into those ArgoCD versions lacks it, so the Application shows every resource as Unknown and syncs no new revision:
Before you move the Kubernetes cluster to 1.37, do one of the following:
- Upgrade ArgoCD to 3.5.0 or later (argo-cd Helm chart 10.2.3 or later), after reading its 3.4 to 3.5 upgrade notes. For Hopsworks three of them matter. Charts render with Helm 4.2, and
helm.version: v3is ignored. An OCI chart registry over plain HTTP needsinsecureOCIForceHttp: "true"in its repository Secret, which has no effect ifinsecure: "true"is also set. An HPA without metrics is now Degraded, so a sync right after a deploy can fail on the KnativeactivatorHPA untilsyncPolicy.retryruns it again; to accept such HPAs, override theautoscaling_HorizontalPodAutoscalerhealth check inargocd-cm. -
Stay on your ArgoCD version and add this to the Application:
Trino query history#
From 5.0.14, 5.1.2 and 5.2.0 the chart no longer enables the Trino MySQL event listener, so hopsworks.trino_queries stops receiving rows, and the administrators' Trino query history shows only what was recorded before the upgrade. To keep it, add the listener in your values:
trino:
trino:
additionalConfigProperties:
- internal-communication.shared-secret=${ENV:TRINO_SHARED_SECRET}
- http-server.process-forwarded=true
- event-listener.config-files=etc/mysql-event-listener.properties
coordinator:
additionalConfigFiles:
mysql-event-listener.properties: &mysqlEventListenerProperties |
event-listener.name=mysql
mysql-event-listener.db.url=jdbc:mysql://${ENV:MYSQL_ADDRESS}/${ENV:MYSQL_DB}?user=${ENV:MYSQL_USER}&password=${ENV:MYSQL_PASSWORD}&useSSL=false&allowPublicKeyRetrieval=true
worker:
additionalConfigFiles:
mysql-event-listener.properties: *mysqlEventListenerProperties
additionalConfigProperties replaces the chart's list, so it must repeat the chart's other entries. Both roles need the file, because the property reaches the workers too.
From the same releases, recorded rows also lose their query text, plan and JSON columns after hopsworks.variables.trino_events_strip_details_after_days (7) days. User, state, timing and resource columns stay until trino_events_delete_after_days (61). The first nightly cleanup after the upgrade strips every existing row older than 7 days. Set the value to 61 or more to keep the details.
Feature groups shared with a subset of their features#
From 5.2.1, because of a security fix, share the whole feature group with a project that queries it in Trino and may read all its features. Shares of a whole feature group, and the online and embedding access of a subset share, are unchanged. hopsworks.variables.trino_feature_group_subset_shares_enabled: "true" keeps the behaviour of 5.2.0, without the fix; keep its default, "false".
consul-server-acl-init fails with field is immutable#
The upgrade can fail on the consul-server-acl-init Job with spec.template: field is immutable. Delete that Job and run the upgrade again.
opensearch-index-upgrade gets no answer from elastic.service.consul#
The upgrade can fail on the opensearch-index-upgrade hook with FATAL: no answer from https://elastic.service.consul:9200, and fail the same way when it is run again. One cause is that the Consul block is missing from the cluster DNS, because something outside the chart has reset the DNS configuration since the last upgrade. An update of the EKS CoreDNS add-on with conflict resolution OVERWRITE does that. Check the Corefile before changing anything:
If it has a consul block, the cause is another one: see No answer from the cluster under If the hook fails. If the block is missing, a chart release with the CoreDNS configuration hook puts it back by itself. With an earlier release, add it at the end of the Corefile, wait a minute for CoreDNS to load it, and run the upgrade again:
This is the form for the default setup, in which the chart configures the cluster DNS through the coredns ConfigMap and sets CONSUL_SERVER on the CoreDNS Deployment.
5.2.x to 5.3.x#
OpenSearch stays on 3.8.0. An upgrade from 5.2.0 or 5.2.1 also gets the changes from 5.2.2. An upgrade from 5.2.0 also crosses two changes of 5.2.1, described under 5.1.x to 5.2.x: Feature groups shared with a subset of their features, and the OpenSearch Security configuration, which addresses a vulnerability.
Additional upgrade prerequisites#
The identity running Helm or syncing ArgoCD must have permission to manage ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding resources in the admissionregistration.k8s.io API group.
Before upgrading, check that every project namespace carries the label configured by global._hopsworks.projectPodSecurity.namespaceLabel, which defaults to hopsworks.ai/project:
If you supply project namespaces yourself, set global._hopsworks.projectPodSecurity.namespaceLabel to an administrator-managed label key, for example ops.example.org/hopsworks-project, and add that label to every project namespace. Include it in hopsworks.variables.project_namespace_labels so Hopsworks also labels new project namespaces, preserving any existing entries in that setting. The hopsworks.ai/project label also controls namespace ownership during uninstallation; do not add it to a namespace you intend to manage yourself.
For multiple releases on the same Kubernetes cluster, set global._hopsworks.projectPodSecurity.namespaceMatchLabels to each release's tenant labels. Leaving namespaceMatchLabels empty selects every namespace carrying namespaceLabel, including projects belonging to other releases. Every release must select a distinct set of project namespaces. Add these labels to existing and pre-created project namespaces and include them in hopsworks.variables.project_namespace_labels before upgrading.
Project pod admission can be turned off with global._hopsworks.projectPodSecurity.enabled: false, which skips these prerequisites, but we recommend keeping it on.
Strimzi 1.3#
From the 5.2.x patch release that uses Strimzi 1.3, and in 5.3, the chart deploys Strimzi 1.3.1 in place of 1.2.0, and Kafka stays on 4.3.1. An upgrade from a release that already uses Strimzi 1.3 crosses none of this.
-
Upgrade central before its satellites, the reverse of the order from 5.1.x to 5.2.x. A satellite runs on central's operator, and the 1.2.0 operator would replace the satellite's controller onto the 1.3.1 image without the configuration that image reads, which leaves the satellite's Kafka down until central is upgraded. The satellite's upgrade therefore refuses to start while its Kafka was last reconciled by an older 1.x operator. Upgrade central, wait until the satellite's Kafka reports the 1.3.1 operator, then upgrade the satellite:
Until its own upgrade, the satellite keeps its pods on the old image under the 1.3.1 operator, and that upgrade replaces each of them once. - No step before helm upgrade. The strimzi-crd-upgrade hook applies the Strimzi 1.3.1 CRDs before the operator is replaced. - The brokers and controllers are replaced once, by the new operator, for the new image. The chart stops the outgoing operator before it starts the new one, through kafka.strimzi-kafka-operator.deploymentStrategy, so that the 1.2.0 operator does not act on the new image. A pod the 1.2.0 operator replaces onto the 1.3.1 image lacks the configuration that image reads, and restarts until the new operator takes over. It then recovers on its own, which can take a few minutes, because Kubernetes waits up to five minutes between restarts of a failing container. The setting makes that unlikely rather than impossible: nothing waits for the old operator to exit. If your values set kafka.strimzi-kafka-operator.deploymentStrategy.type: Recreate, also set rollingUpdate: null there, or the merged strategy is rejected. A single-broker cluster is unavailable while its broker is replaced, a single-controller cluster has no KRaft quorum while its controller is replaced, and OnlineFS catches up afterwards. - Three new tags to mirror on an airgapped cluster: strimzi/operator, strimzi/kafka and strimzi/crds, which vendor_images.sh lists. - The operator deletes pods with background propagation during a rolling update. Strimzi 1.3 turns its UseBackgroundPodDeletion feature gate on by default, so that CNI plugins which remove a pod's network endpoints when its object is deleted, such as Cilium, no longer cut a broker off before it has shut down. To keep the earlier foreground deletion, set kafka.strimzi-kafka-operator.featureGates: "-UseBackgroundPodDeletion". - Under ArgoCD, the ignoreDifferences entry for the Kafka CRD is no longer needed. Strimzi 1.3.1's Kafka CRD no longer has the empty map that the entry in Under ArgoCD ignores, and the entry matches nothing on it, so it can stay.
CoreDNS configuration hook#
From the 5.2.x patch release that carries this hook, and in 5.3, an upgrade configures the cluster DNS for the Consul domain before its other hooks run. The other hooks reach services by their Consul names, so an upgrade used to fail on its first hook when the Consul block was missing from the cluster DNS, as described under opensearch-index-upgrade gets no answer. The hook is rendered when consul.autoConfigureCoreDNS is on, which is the default.
- What runs. The
consul-coredns-pre-upgradehook runs the same step as theconsul-coredns-autoconfJob, which still runs after the release is applied. It then waits untilconsul.service.consulhas resolved 20 times in a row, for at most 3 minutes, because the CoreDNS pods load a changed configuration one by one. On a cluster whose DNS is in order it changes nothing and adds about 20 seconds. - The hook does not fail the upgrade by itself. It works with the roles of the previous release, so it cannot follow a change of
consul.corednsConfigMapNameorconsul.corednsDeploymentNamemade in the same upgrade; theconsul-coredns-autoconfJob applies that change afterwards. When the hook cannot read or write the DNS configuration, or the name does not resolve in time, it says so in its log and the upgrade continues. Its Job gives up after 4 minutes, under the 5 minutes Helm waits for a hook by default. -
The log says what the hook did. The Job is kept after it finishes, for one day by default, so that its log can be read:
Its lines start with >> coredns pre-upgrade hook:.
| Line | Meaning | What to do |
| ---- | ------- | ---------- |
| `consul.service.consul resolved 20 times in a row` | The hook did its work. | Nothing. |
| `could not read the Service` or `does not exist yet` | The hook could not find the Consul DNS Service and left the DNS configuration alone. | Nothing, unless a later hook fails on a Consul name: then check that Service and run the upgrade again. |
| `configure-dns.sh ended with status` | The hook could not read or write the DNS configuration; the error is above that line. | Nothing in an upgrade that changes one of the two names above. Otherwise check the Role `consul-autoconfigcoredns-update-dns-role` in `kube-system` and those two values. |
| `did not resolve 20 times in a row` | Consul names do not resolve reliably yet. | If a later hook fails on a Consul name, check the endpoints of the Consul DNS Service, the Consul server pods and the logs of the CoreDNS pods, then run the upgrade again. |
-
When the hook's Job gets no pod. With
global._hopsworks.mode: upgrade, as under ArgoCD, the chart cannot see whether the service account of the hook exists. Turningconsul.autoConfigureCoreDNSon for an existing installation in that mode creates the Job before its service account: the Job gets no pod, its events sayerror looking up service account hopsworks/consul-autoconfigcoredns-account: serviceaccount "consul-autoconfigcoredns-account" not found, and the upgrade or sync fails on the hook after 4 minutes. Helm reportspre-upgrade hooks failedwithjob consul-coredns-pre-upgrade-<hash>-r<revision> failed: DeadlineExceeded, and nothing else of the release has been applied at that point. The Job carries the annotationhopsworks.ai/no-podfor this case. Under ArgoCD, apply the service account, the two Roles and the two RoleBindings of the Job first, with the values of the application, then sync:helm template hopsworks hopsworks/hopsworks --version <version> --namespace hopsworks --values <values> \ --show-only charts/consul/templates/consul-autoconfigcoredns-ServiceAccount.yaml \ --show-only charts/consul/templates/consul-autoconfigcoredns-role.yaml \ --show-only charts/consul/templates/consul-autoconfigcoredns-rolebinding.yaml \ | kubectl apply -f -
With Helm, run that upgrade without global._hopsworks.mode, so that it takes its default, auto: the chart then sees that the service account is missing and leaves the hook out. Applying the objects by hand does not work with Helm, which refuses objects it did not create.
ICEBERG feature groups in Trino#
From 5.3, the Trino iceberg catalog reads an ICEBERG feature group on HopsFS once Hopsworks has registered its current Iceberg metadata in the Hive metastore. Hopsworks does that on every commit, and when the client syncs after an empty save or a schema change, so a feature group created before the upgrade is readable from Trino after its next write.
To register every existing ICEBERG feature group at once, call the backfill endpoint as an administrator, passing the returned nextAfterId as afterId, until it is null:
Each item of the response reports one feature group:
PUBLISHED: registered now.UP_TO_DATE: already registered.NO_METADATA: nothing committed yet.SKIPPED: not mirrored, or deleted.FOREIGN_LOCATION: its metastore table is not at the feature group's location, so it is not registered.FAILED: with the error.
The call is safe to repeat. Each feature group costs a few HopsFS reads and one metastore update, and there is no outage.
Two Trino changes come with it:
- The
hivecatalog no longer reads ICEBERG feature groups, because the Trino Hive connector refuses tables registered as Iceberg. Queries and Superset datasets that read them throughhivemust move toiceberg. - In the
icebergcatalog the feature store schema is read-only, for data owners too.
Spark access by name (spark.table, spark.sql, DESCRIBE) to an ICEBERG feature group needs the Iceberg storage handler on the Spark classpath, which the base images carry from 5.2. Custom Python environments are not rebuilt on upgrade: a project environment cloned from a release before 5.2 keeps its own image, built from that release's base. Once a feature group is registered as table_type=ICEBERG, at its next commit or by the backfill, access by name from such an environment fails with Failed to load storage handler until the environment is rebuilt. Installing or uninstalling a library rebuilds the environment from the current base image. Reads and writes by path through the feature store client are not affected.
Filebeat registry#
Releases before 5.3 mounted the Filebeat registry at a path Filebeat never wrote to, so every Filebeat restart, pod re-creation and helm upgrade shipped all of a node's current container logs again into .services-* and the project log indices. On the upgrade to 5.3:
- The registry moves to
/var/lib/filebeat-data/<namespace>on each node. That directory starts empty, so each node ships its current container logs once more, as every restart did before. Later restarts and upgrades resume from the registry. - Leave the existing contents of
/var/lib/filebeat-datain place. Nothing in the chart reads them, and they may belong to another Filebeat on the node.
Logstash 9#
In 5.3, and from the 5.2.x patch release that ships it, Logstash moves from 7.17 to 9.5. The chart's own pipelines need nothing, and documents keep their field names, because the chart sets pipeline.ecs_compatibility: disabled. Pipelines, outputs and plugins you added through logstash.extraPipelines, logstash.additional_audit_outputs or logstash.plugins need a check before the upgrade:
- Rename the TLS settings of
elasticsearch,httpandtcpoutputs in the same upgrade. The output plugins move up a major version:elasticsearch11 to 12,http5 to 6,tcp6 to 7 andopensearch1 to 2. The newelasticsearch,httpandtcpoutputs refuse their old SSL settings, such ascacert,sslandssl_certificate_verificationonelasticsearch,cacert,client_cert,client_key,keystoreandtruststoreonhttp, andssl_enable,ssl_cert,ssl_cacertandssl_verifyontcp. The 7.17 versions do not know the replacement names, such asssl_enabledandssl_certificate_authorities, so no TLS setting works on both versions, and the values of the upgrade itself must carry the new names. Each plugin's page in the Logstash plugin reference lists them. - A pipeline that fails to start stops all log shipping. Logstash keeps running the other pipelines, but the routing pipeline waits on every event for the one that did not start, so service, project and audit logs all stop while the Logstash pod stays ready. After the upgrade, look for
is obsolete and is no longer availablein the Logstash log. - The kafka integration, the
azure_event_hubsinput and thewebhdfsoutput are no longer in the image. A pipeline that uses one of them needs it inlogstash.plugins, which installs it when the Logstash pod starts, so it needs access torubygems.organd cannot run on an airgapped cluster.
Agent deployments reserve 2 GiB for their OTLP sidecar#
An agent deployment with tracing enabled runs a hops-otlp sidecar next to the agent. hopsworks.variables.agent_deployment_otel_memory_mb is both the memory request and the memory limit of that container, so the value is reserved on the node, not only allowed. From 5.3 the default is 2048, up from 1024, because at 1 GiB the sidecar was OOM-killed under ordinary traffic.
- Check capacity before you redeploy. A cluster that runs N traced agent deployments reserves 2 GiB x N for their sidecars, where it reserved 1 GiB x N. On nodes sized for the old default, an agent pod recreated after the upgrade can stay
Pendingfor lack of memory. Count the traced agent deployments against the free memory of the nodes that schedule them before you redeploy them. - Running pods keep the old reservation until they are recreated. The upgrade does not restart agent deployments, so nothing changes for a running one until it is stopped and started, saved, or rescheduled. The increase arrives one deployment at a time, which is also why a shortage shows up later than the upgrade.
- Keeping 1024 keeps the OOM.
hopsworks.variables.agent_deployment_otel_memory_mb: "1024"pins the old value on a cluster that cannot spare the memory, at the cost the increase removes. The buffering fixes in the 5.3 sidecar image make an OOM much less likely than before, because what the sidecar holds no longer grows with how far the feature store write falls behind, but it remains the trade. - What an OOM looks like.
hops-otlpis an ordinary second container without probes, so the OOM kill restarts that container alone: the agent container'srestartCountstays 0, and only the sidecar'slastStateshowsOOMKilled. While the sidecar restarts, the pod reportsReady=Falseand leaves the endpoints of its service, so the agent, healthy and still running, receives no requests. Looking for agent restarts finds none.
Superset 6.1#
5.3 moves Superset from 6.0.0p4 to 6.1.0. The hopsworks-superset-init-db Job runs Superset's schema migrations on the existing metadata database once, during the upgrade, and the Superset pods wait for it. On a cluster upgraded from a 6.0.0p4 database it ran 6 migrations in about a minute and kept every database connection and dashboard.
- Back up Superset first. Run the backup before the upgrade:
kubectl create job -n hopsworks --from=cronjob/create-superset-backup superset-backup-pre-53. The migrations have no downgrade, and the restore Job loads a dump into whatever image the release runs. To go back to a 6.0 backup, restore it with the 5.2 chart, whose image matches it, then upgrade again. - The Celery worker needs more memory. The worker's request and limit go from 1 GiB to 2 GiB. Check that a node in the namespace has 2 GiB free for it.
- The MCP service runs. A new
hopsworks-superset-mcpDeployment serves thehops superset chartcommands over HTTPS, with a certificate from the cluster's CA and a signing key in the Secretsuperset-mcp-jwt. Under ArgoCD, setsuperset.mcp.jwtSecret, or the service is not deployed; see MCP service. - Semantic layer dashboards. With the semantic layer enabled, each Data owner and Data scientist of a project with Superset and a semantic project gets one Superset database connection, which reads the metrics of the project's semantic projects, see Dashboards in Superset. Set
hopsworks.variables.superset_semantic_enabled: "false"to keep that off.
Semantic layer#
5.3 adds the semantic layer, off by default. Before turning it on with global._hopsworks.semanticLayer.enabled: true:
- Room in
ts_1. The query history is the tablehopsworks.semantic_queryin RonDB's disk tablespacets_1, which feature groups with disk columns share, and keeps each query for 30 days at about 13,000 bytes. Check the free space ofts_1and add a data file when the history will need it, see Query history storage. - The 5.3 library in
python-feature-pipeline. Metric export jobs run in each project'spython-feature-pipelineenvironment, and fail before they write with an older Hopsworks library there, see Who runs an export. - Memory for the gateway. Each of the two semantic gateway replicas takes 4 GiB, see Memory.
Kueue outside the Hopsworks namespace#
Required for a cluster that runs the Kueue bundled with Hopsworks (global._hopsworks.kueue.enabled=true); a cluster without Kueue needs nothing. From the 5.2.x patch release that uses Kueue 0.20, and in 5.3, Hopsworks uses a Kueue installed as its own release, kueue in kueue-system, in place of the Kueue 0.12.2 bundled in the Hopsworks namespace: the chart ships no Kueue. The new chart refuses to upgrade a release that still runs the bundled Kueue 0.12, and a cluster that runs it cannot be switched with helm upgrade alone: Kueue 0.12 stores Cohorts and Topologies as v1alpha1, which no later Kueue reads. The 5.1 chart's migrate-to-external.sh moves it to Kueue 0.18.1 in kueue-system and keeps every queue, on 5.1, or on 5.2.0 and 5.2.1 together with the Hopsworks upgrade. After the Hopsworks upgrade, the new chart's upgrade-kueue.sh upgrades Kueue to 0.20.3. Follow Kueue outside the Hopsworks namespace, and stop submitting jobs while the move runs.
What changes for users and operators (details):
- Spark jobs are queued, with static executors: a job saved with dynamic allocation runs its dynamic-allocation minimum, and
spark.dynamicAllocation.enabled=truein its Spark properties is refused. - A workload whose pods are not all ready 30 minutes after admission is evicted and queued again (Kueue 0.19
waitForPodsReady). - The Ray autoscaler and job-submitter sidecars count against quota (Kueue 0.19).
- A workload without a priority class gets the WorkloadPriorityClass named
default, when one exists (Kueue 0.20). - Disk limits need Kueue to leave
ephemeral-storageout of its quota checks, which the upgrade script sets up. It stops before changing the cluster when it cannot read the queues, or when disk is budgeted but some ClusterQueues do not cover it (Disk limits).
Kafka node pool settings apply on every upgrade#
From 5.2.5, and in 5.3, a Helm upgrade applies the Kafka node pool settings in your values again. On a cluster whose Strimzi CRDs serve only v1, which is the case for a cluster installed on 5.2, earlier 5.2 releases left the two KafkaNodePool resources as they were on every Helm upgrade. A change of the replicas, storage, resources, JVM options or placement of the brokers or controllers had no effect, and a deleted pool did not come back. The kafka-kraft-migration pre-upgrade hook now applies both pools on every Helm upgrade. Under ArgoCD, and with Helm when global._hopsworks.mode is set, the pools are part of the release and are applied with it, so they were not affected, and the hook still leaves them to the release.
-
The first upgrade with the fix applies what your values say. That includes settings that earlier upgrades ignored, and it overwrites a change made to a pool by hand, such as a replica count set with
kubectl, because the hook applies every field the chart renders. Compare the live pools with your values before upgrading:
Strimzi replaces the Kafka pods one by one for a change of placement, resources or JVM options, and adds brokers for a higher replica count. - A setting removed from the values can stay on a pool. The hook applies the pools server-side as the field manager kafka-kraft-migration, and Kubernetes removes a field that the values no longer set only when that manager is its only owner. Helm owns the fields it set at install, and keeps owning each one whose value no upgrade has changed since, so such a field stays on the pool. Check the pool after removing a setting, and remove the field from the pool by hand if it is still there. - kafka.migrationJob.enabled stays on. The hook is the only thing that applies node pool settings during a Helm upgrade without global._hopsworks.mode, so with the switch off, pool changes are not applied. - The upgrade fails when the pools cannot be applied. When the hook cannot read the KafkaNodePool CRD or the pools, or the cluster rejects the apply, the upgrade fails on the kafka-kraft-migration hook, where it used to continue without applying anything. The log of the hook has the error:
```sh
kubectl logs -n hopsworks job/kafka-kraft-migration
```
- When the hook leaves the pools alone. When
v1beta2has been removed from theKafkaNodePoolCRD while a pool still carries ownership records written at that version, the cluster rejects every server-side apply to that pool. The hook then applies nothing, and its log saysNot applying the node pools in this upgradeand names the pools. Pool changes are applied by the first upgrade after those records are cleared. Thestrimzi-crd-upgradehook clears them later in the same upgrade when it is enabled (kafka.crdUpgradeJob.enabled), the chart runs the Strimzi operator (kafka.strimzi-kafka-operator.enabled), and every Strimzi CRD stores onlyv1. Otherwise whoever manages the Strimzi CRDs has to clear them, and until then every upgrade leaves the pools as they are.
Bring your own Kafka: Hopsworks creates and deletes topics#
From 5.3, with enable_bring_your_own_kafka, Hopsworks creates and deletes the topics it needs on the external Kafka, as it does on the bundled one. It connects with each feature store's kafka_connector. Before, it created no topics there, so every topic had to exist before ingestion.
- The backend connects to the external brokers. The operations below make the Hopsworks pods call the brokers of the project's
kafka_connector, also for a topic that already exists, so those brokers must be reachable from the Hopsworks pods with the connector's settings. Otherwise the operations fail withKAFKA_UNAVAILABLE(190021) after 30 seconds,TOPIC_FETCH_FAILED(190023) when the broker refuses to describe the topic, orKAFKA_GENERIC_ERROR(190007) when the client cannot be set up. - Topics that already exist are used as they are. Hopsworks describes a topic before creating it, and never deletes a topic it found already there. Topics created before the upgrade keep working with the same
kafka_connectorcredentials, which already need Describe on them to produce. - Missing topics are created. Creating an online or stream feature group, changing a feature group's topic or notification topic, setting the project topic in
Project Settings→Kafka, or creating a topic through the topic API creates the topic on the external cluster when it does not exist. A stream feature group that is not online-enabled writes to a topic named after the project, with no suffix, unless the project or the feature group names one. For that thekafka_connectorprincipal needs Create on those topics, and Delete for Hopsworks to remove them again. Without Create, these operations fail withTOPIC_CREATION_FAILED(190024) and the broker's reason; before 5.3 they succeeded and ingestion failed later on the missing topic. - Hopsworks deletes only the topics it created, with their feature group or project, or when a feature group moves to another topic, and only those no other project used when they were created. A feature group's topic also stays while another feature group names it or while it is the project topic. A topic Hopsworks created on the external cluster is deleted by name on the cluster the project's
kafka_connectorpoints at when it is deleted. A delete that Kafka refuses keeps the topic in the project's topic list; one refused while deleting a project is logged by the backend with the topic's name and cluster. - Topics recorded before the upgrade stay on the bundled Kafka. Before 5.3 Hopsworks created every topic it recorded on the bundled Kafka, also for projects that had no
kafka_connectoryet. The upgrade adds the nullable columnproject_topics.external_kafka(migration V130, an onlineALTER), and a row without it is deleted on the bundled Kafka, as before, never on the external cluster. A project without akafka_connectorstill creates its topics on the bundled Kafka, so create the connector before the project's first feature group. - Partitions and replicas. The project topic and the online and stream feature group topics get
hopsworks.variables.onlinefs_service_thread_numberpartitions (default 10) andhopsworks.variables.kafka_num_replicasreplicas (default 1); topics created through the topic API get the partitions and replicas they ask for. Wheremin.insync.replicasis above a topic's replicas, producers that write withacks=allfail withNotEnoughReplicas, so setkafka_num_replicasto the external cluster's replication factor. The variable applies to every project: one without akafka_connectorcreates its topics on the bundled Kafka, which refuses more replicas than it has brokers withTOPIC_REPLICATION_ERROR(190005). - OnlineFS must subscribe by pattern. OnlineFS picks up new topics only through
onlinefs.configmapOverrides.onlinefssite.kafkaConsumer.topicPattern(default.*_onlinefs). A non-emptytopicListin the same section replaces the pattern and does not include the topics Hopsworks creates. A cluster set up with a topic list setstopicList: "", and widenstopicPatternuntil it also matches the project topics it consumes that do not end in_onlinefs. - Model serving does not log to an external Kafka. The inference logger reaches only the bundled Kafka, so a deployment created with inference logging gets no topic, and selecting an existing topic for it is refused.
Notification topics and shared feature stores#
- A notification topic belongs to its project. A feature group's notification topic (Change Data Capture) is a topic of its project from 5.3, like its ingestion topic. Hopsworks creates it when the feature group is created or moves to another notification topic, if it does not exist yet, and deletes it with the feature group, or on such a move, when Hopsworks created it and no other feature group uses it. A feature group whose notification topic is another project's topic is refused with
TOPIC_IN_USE_BY_ANOTHER_PROJECT(190026). A notification topic set before the upgrade is left as it is until its feature group moves to another one. - Kafka changes through a shared feature store are refused. Creating or deleting a topic or a schema, or setting a compatibility level, through a feature store shared from another project fails with
CHANGE_THROUGH_SHARED_FEATURE_STORE(190027). Reading through a shared feature store is unchanged.
Deleting a tag schema in use is refused#
Up to 5.2, deleting a tag schema deleted every tag created from it, in every project. From 5.3, the delete is refused with TAG_SCHEMA_IN_USE (370014, HTTP 409) while a tag created from the schema is attached to any artifact, or while the schema has archived values in the tag history. The message says how many tags use the schema, per kind of artifact, or how many archived values it has.
- Automation that deletes tag schemas has to remove their tags first, with
delete_tag(). A schema with archived values cannot be deleted, because the history keeps them. -
Wait for the dataset tag migration before deleting a tag schema. Legacy dataset tags stored in HopsFS are moved into the database by a background migration that starts 30 minutes after Hopsworks starts, and the delete does not see tags that have not moved yet. A dataset whose legacy tag names a deleted schema fails the migration and keeps its tags in HopsFS. The migration has finished when the
dataset_tags_xattr_migration_donesetting istrue. -
A deleted schema's name can stay taken. A schema deleted before 5.3 kept its archived values in the tag history, and a new schema of the same name would take them over. Creating a schema under such a name is refused with
TAG_SCHEMA_NAME_HAS_HISTORY(370015, HTTP 409), so choose another name.
The extended attributes endpoint is removed#
From 5.3, /project/{id}/xattrs/{path} answers 404: HopsFS extended attributes can no longer be read, written or deleted through the REST API. Tags on datasets, feature groups, models and the other artifacts are managed through their tag APIs, as before.
A dataset whose legacy tags cannot be moved into the database answers tag changes with TAG_MIGRATION_PENDING (370013, HTTP 409), and the Hopsworks log names the dataset and the tags that failed. A tag fails when its tag schema was deleted or its value no longer validates against the schema. Re-creating a deleted schema clears the dataset when its values validate. Otherwise an administrator edits the dataset's user.tags extended attribute as the HopsFS superuser, which the NameNode container runs as. The attribute holds a JSON array with one {"key": <tag schema>, "value": <tag value as a JSON string>} entry per tag.
-
Read the attribute, with the dataset's HopsFS path, for example
/Projects/<project>/<dataset>:kubectl exec -n hopsworks deploy/namenode -c namenode -- \ /srv/hops/hadoop/bin/hdfs dfs -getfattr -n user.tags -e text /Projects/<project>/<dataset>The value is the text between the outer quotes:
-
Write it back without the tags that failed, or with their values corrected:
kubectl exec -n hopsworks deploy/namenode -c namenode -- \ /srv/hops/hadoop/bin/hdfs dfs -setfattr -n user.tags \ -v '[{"key":"pii","value":"{\"owner\":\"ann\"}"}]' /Projects/<project>/<dataset>When no tag should be kept, remove the attribute with
-setfattr -x user.tagsinstead.
The dataset's tags move into the database the next time they are read, or at the hourly migration pass.
Envoy Gateway#
New in 5.3 and off by default: Envoy Gateway routes Standard-mode model deployments through the Kubernetes Gateway API, with Istio still the front door.
- Nothing moves unless you enable it. An upgrade that leaves
global._hopsworks.envoyGatewayalone routes every request as before. - Two components restart during the upgrade. The KServe controller restarts, and the running deployments keep serving meanwhile. From 5.3 it also restarts on any later upgrade that changes its ingress configuration, such as turning Envoy Gateway routing on or off.
model-serving-authenticatorrestarts with a new listener for the Gateway. While its old pod is replaced, a few requests can get a 403, because the 5.2 pod stops without waiting for the new one; from 5.3 its restarts wait, so later ones lose no request.
To enable it in a cluster with running deployments, upgrade twice, so that the Gateway is up before any deployment moves. Clients keep their endpoint, Knative-mode deployments stay on Istio, and no deployment restarts.
-
Prepare. List the Standard-mode deployments, to check them at the end:
kubectl get isvc -A -o json | jq -r '.items[] | select(.metadata.annotations["serving.kserve.io/deploymentMode"] == "Standard") | .metadata.namespace + "/" + .metadata.name'OpenShift 4.19 or later, and GKE
Set
hw-envoy-gateway.gatewayApiCrds.enabled: falsebefore you turn the feature on. The platform owns the Gateway API CRDs there, and the chart must use them rather than install its own, as described in CustomResourceDefinitions.values.openshift.yamlalready sets it. On GKE Standard, enable the Gateway API on the cluster first: GKE installs the CRDs only then, and the upgrade fails without them.With
rbac.useExistingRole, add the permissions listed under Cluster requirements to the backend role. -
Install Envoy Gateway with
helm upgradeand these values:Set
kserveStandardMode: falseexplicitly: it defaults totrue, which would also move the deployments in this upgrade. The deployments keep their Istio routes, so no traffic moves yet. The Gateway is ready when it reportsProgrammed:Hopsworks checks the rest itself before it moves a deployment: the Gateway's policy, the routes and a request through Envoy. You can stay at this step for as long as you need.
-
Move the deployments with
helm upgradeandkserveStandardMode: true. The KServe controller restarts, and Hopsworks starts moving the Standard-mode deployments within a minute. Each one moves once its new routes are accepted, and its Istio route keeps serving until then; a deployment still waiting is retried every minute. A deployment that is not ready is picked up after it becomes ready, at the next full pass, every ten minutes. The backend log reports each deployment it moves. -
Check against the list from step 1:
-
No Istio Ingress is left for a ready Standard-mode deployment, neither KServe's nor, for a deployment that scales to zero, the idle one:
-
Every Standard-mode deployment answers through the usual endpoint, with both URL shapes.
- The Knative-mode deployments are unchanged.
-
Headers with underscores
Once a deployment has moved, the Gateway drops request headers whose names contain an underscore, such as session_id, which Istio passed on. The request still succeeds, but a predictor or transformer script that reads such a header no longer receives it. Send the value in a header whose name uses hyphens instead.
OnlineFS offsets move to onlinefs.kafka_offsets#
From 5.3, OnlineFS keeps its consumer offsets in one table, onlinefs.kafka_offsets, in the RonDB it writes to, with the ID of the Kafka topic each offset was saved for, in place of a kafka_offsets table in the online feature store of each project. With the topic ID, OnlineFS reads a topic that was deleted and created again under the same name from its start, as after a restore that rebuilt Kafka, instead of resuming at an offset of the earlier topic (Consumer offsets).
- No step before
helm upgrade. Each OnlineFS pod runs anoffsets-schemainit container before OnlineFS starts. It creates the table, copies the offsets of every project into it, and lists each copied project database inonlinefs.kafka_offsets_moved. For each consumer group, topic and partition it keeps the highest offset the project tables hold, with the topic's current ID, and leaves out the offsets of topics Kafka no longer has. A pod that starts later copies only project tables not listed yet. - The project tables stay until every OnlineFS pod runs 5.3. An OnlineFS pod of 5.2 reads its project's table each time Kafka gives it partitions, as during the rollout. The
onlinefs-drop-legacy-offsets-<hash>-r<revision>Job waits for the rollout of the OnlineFS Deployment to finish, then drops the listed project tables. On a cluster withglobal._hopsworks.full_platform: trueit holds the Flyway lease while it drops them, as the schema migrations do. - The init container and the Job connect to the DDL SQL node of the release as
hopsworksroot, as the schema migrations do: theddl-mysqldService of the release, or, when RonDB is managed externally on a release withglobal._hopsworks.full_platform: true, the consul name inonlinefs.dependencies.mysql.ddlConsulServiceName(mysqlddl). A satellite cluster (global._hopsworks.full_platform: false) always connects to the Service, also with an external RonDB. A cluster that disables the DDL SQL node (rondb.rondb.meta.ddlMySQLd.enabled: false) setsonlinefs.dependencies.mysql.serviceName: mysqld, and alsoonlinefs.dependencies.mysql.ddlConsulServiceName: mysqlwhen it is not a satellite and RonDB is managed externally.onlinefs.dependencies.mysql.urloverrides both. - With
onlinefs.rbac.useExistingRole, the role needsget,listandwatchondeployments, and on a cluster withglobal._hopsworks.full_platform: truethe lease rules of the chart'sonlinefs-role, for the Job. -
A step that fails keeps the pod from starting. It reads every project table before it writes, so a step that fails while it copies leaves every project table as it is, and the next start copies them again. The pod stays in
Init:ErrororInit:CrashLoopBackOff, and the log names the database:
The step gives up after onlinefs.offsetsSchemaBudgetSeconds, 900 seconds by default, its wait for Hopsworks to answer with the OnlineFS settings included, and Kubernetes starts it again. Fix the cause, as described in If the offsets-schema step fails, and the next start of the pod runs it again. - Records written during the rollout can be written to the online feature store twice, never skipped. A new OnlineFS pod resumes a partition from the offset copied when the first new pod started, so it writes again the records the pods of the earlier release processed since. - On satellite clusters, upgraded after central as Strimzi 1.3 requires, a project created after central's upgrade has no offsets table on a satellite that is not upgraded yet. The satellite's OnlineFS logs Failed to save offsets for that project and keeps writing its rows, but reads the project's online topics again from their start when it restarts, and once more at the satellite's upgrade. - Rolling OnlineFS back to 5.2 after the Job dropped the project tables makes it read every online topic again from its earliest retained record, because the tables it reads are gone. Before the Job ran, it resumes from the project tables, at the offsets they held when the 5.3 pods started. An offset set in Kafka offsets that OnlineFS had not applied yet waits in onlinefs.kafka_offsets, and applies once OnlineFS is upgraded again if the topic was not created again meanwhile and Kafka still has the offset. - A restore needs no step for the offsets on a Kafka that gives its topics IDs, as the Kafka the chart installs does. onlinefs.kafka_offsets is in the RonDB backup like any other database, and after a restore that rebuilt Kafka, OnlineFS reads the recreated topics from their start. An offset saved without a topic ID, by a consumer group of an earlier Kafka connection or on a Kafka that gives no IDs, has nothing to check against, and OnlineFS resumes from it.
If the offsets-schema step fails#
A step that fails logs Failed to set up onlinefs.kafka_offsets, OnlineFS does not start, followed by the error, in the offsets-schema container of a pod that is not Running:
| The error | What it means | What to do |
|---|---|---|
Hopsworks did not answer with the OnlineFS settings within 900 s | Hopsworks did not answer within the step's time, onlinefs.offsetsSchemaBudgetSeconds. Nothing was changed. | Wait until the hopsworks-instance pods are ready. The next start runs the step again. |
Communications link failure, UnknownHostException or Access denied for user 'hopsworksroot' | The step cannot reach or log in to the SQL node. Nothing was changed. | While the upgrade restarts the SQL node, this passes on its own. Otherwise check the SQL node the step connects to, as described above, and the hopsworksroot key of the mysql-users-secrets Secret. |
Got error 4009 'No data node(s) available' | RonDB's data nodes are restarting or down, as while the upgrade restarts them. Nothing was changed. | The next start runs the step again once the data nodes are up. |
onlinefs.kafka_offsets has the columns [...], OnlineFS maps [...] | The table exists with columns OnlineFS does not map, because it was created or changed outside OnlineFS. Nothing was moved. | Do not drop it, it holds the offsets. Change its columns to the ones the message lists after OnlineFS maps, keeping its rows. |
Failed to read a per-project kafka_offsets, it is kept as it is, with the database | The table of that project cannot be read. A single such table stops the whole step: nothing was written, and every project table is kept. | Repair that table, from the RonDB backup if needed. A table that cannot be repaired can be dropped, as described below. |
A Kafka error, such as org.apache.kafka.common.errors.TimeoutException | The step could not ask Kafka for the IDs of the topics. Nothing was written. | Check that OnlineFS reaches Kafka, as for its own consumers. |
onlinefs.kafka_offsets was not set up within 900 s | The step ran out of time while it copied the offsets. A project table counts as copied only once every offset is written, so the next start reads every project table and writes every offset again, and needs as long. | Raise onlinefs.offsetsSchemaBudgetSeconds and run helm upgrade. Offsets written already keep their values. |
The Job logs each table it drops, and Dropped <n> per-project kafka_offsets, none is left when it is done:
kubectl get jobs -n <namespace> | grep onlinefs-drop-legacy-offsets
kubectl logs job/<job> -c drop-legacy-offsets -n <namespace>
It fails, and drops no table that is not listed, when a project table was not copied, as while OnlineFS is scaled to 0, or a DROP TABLE statement fails. Until a Job drops them, the tables stay, and each OnlineFS pod logs them as waiting for the drop Job when it starts. The Job of the next helm upgrade runs again; the tables can also be dropped by hand once every OnlineFS pod runs 5.3, as below.
The commands below run SQL as the RonDB root user on the chart's mysqlds-0 pod; with an external RonDB, run the statements on one of its SQL nodes instead. A project table that cannot be repaired can be dropped, so that the step moves the others:
NAMESPACE=hopsworks
ROOT_PW=$(kubectl get secret mysql-users-secrets -n "$NAMESPACE" -o jsonpath='{.data.root}' | base64 -d)
kubectl exec mysqlds-0 -n "$NAMESPACE" -c mysqld -- env MYSQL_PWD="$ROOT_PW" mysql -uroot -e \
'DROP TABLE `<database>`.kafka_offsets'
OnlineFS then has no offsets for the online topics of that project and reads them from their earliest retained record, so it writes the records Kafka still retains to the online feature store again, and skips none.
The move has finished when every OnlineFS pod is Running, the Job has completed, and no project table is left:
kubectl exec mysqlds-0 -n "$NAMESPACE" -c mysqld -- env MYSQL_PWD="$ROOT_PW" mysql -uroot -N -e \
"SELECT table_schema FROM information_schema.tables WHERE table_name = 'kafka_offsets' AND table_schema <> 'onlinefs'"
Kafka offsets in Cluster Settings → OnlineFS then shows the moved offsets as the current offset of each partition.
Search index rebuild#
After the upgrade, the post-upgrade-opensearch-reindex-job hook indexes the existing data sources, which 5.3 adds to search, and datasets, which a cluster upgraded from 5.1 may lack, without emptying the index. No action is needed unless hopsworks.opensearchReindex.enabled is false or the hook's log reports a run already in progress: then request a reindex with scope=ADDITIVE through the REST API once no run is in progress.
LLM Gateway#
The LLM Gateway is new in 5.3 and off by default. On top of Envoy Gateway, global._hopsworks.envoyGateway.aiGateway.enabled gives every project one OpenAI-compatible endpoint, <inference endpoint>/v1/<project>/v1/..., for its Standard-mode vLLM deployments. It brings the LLM Gateway controller, its six CRDs and a dedicated AI Gateway, a Redis and Envoy Gateway's rate-limit service for the rate limits, an access log on the AI Gateway's Envoy, and Grafana dashboards. Tracing has a switch of its own, aiGateway.tracing.enabled, also off by default.
An upgrade that leaves aiGateway alone changes nothing: the Istio Lua filter is the same, no object is added, and the backend finds no envoy-ai-gateway-config ConfigMap and does nothing. Rendering fails when aiGateway.enabled is on and envoyGateway.enabled is off.
Before enabling, as listed under How to enable it:
- Stop the started deployments named
v1, or recreate them under another name. Once the feature is on, such a deployment answers404through the Istio ingress and is not served by the LLM Gateway; it keeps answering through the Envoy Gateway. - Check cert-manager, a network plugin that enforces NetworkPolicies, and the backend's grants in the release namespace.
- On an airgapped cluster, mirror the four images.
Enable with one helm upgrade: no predictive pod, Knative pod or kserve-ingress-gateway pod restarts, and the AI Gateway's Envoy pods roll once when the first route attaches. Data Owners can then turn the LLM Gateway off per project and per deployment, both on by default. Disable with one upgrade with aiGateway.enabled: false, as described under Turn it off, which also lists what a helm rollback leaves behind and how to remove it.
Security response headers#
Hopsworks now sets security response headers itself, including a Content-Security-Policy that the UI enforces. An upgrade needs no action. The policy allows what the Hopsworks UI loads from the Hopsworks origin only. If the UI calls model deployments on another host or port, or another site frames the UI, add that origin to hopsworks.payara.securityHeaders.extraSources, as described under Model deployments on another host or port and Embedding Hopsworks in another site. A page the policy breaks shows a Content-Security-Policy error in the browser console; set cspMode: report-only while finding what it needs.
Pages are no longer served from the browser's offline cache, so a page needs the cluster to be reachable.
Istio 1.31#
Istio moves from 1.30 to 1.31. Istio upgrades in place one minor release at a time, so upgrade to 5.3 from 5.2.5 or later, the release that moves Istio to 1.30, as described in Istio 1.30. On a cluster still on Istio 1.29, the Istio install hook stops with an error before it changes anything; upgrade to the latest 5.2 release first.
- Time. Each wait in the Istio install hook, the install and then istiod and each gateway rolling out, is bounded by
kserve.istio.istioctl.readinessTimeout(15 minutes), so allow at least an hour for the hook inhelm upgrade --timeout. - Model deployments. The Istio ingress gateway and the cluster-local gateway restart once. Each rolls out its new pod before it stops the old one.
- Your own image registry. Mirror
istio/pilotandistio/proxyv2at1.31.4before the upgrade. The chart'svendor_images.shlists both. - Sidecars you injected yourself. The chart injects no Istio sidecar. A workload you gave an Istio sidecar keeps its 1.30 proxy until it restarts, so restart it after the upgrade.
- Going back to 5.2. A
helm rollbackdoes not run the Istio install hook, so Istio stays on 1.31. A laterhelm upgrade, or an ArgoCD sync, with 5.2.5 or a later 5.2 chart stops in that hook before it changes anything, since 5.2 does not install Istio over 1.31. Do not runhelm upgrade, or an ArgoCD sync, with a release older than 5.2.5 afterwards: its hook installs Istio 1.29 over 1.31, a downgrade Istio does not support. To go back entirely, restore the backup on a new cluster, as described in New Cluster Restore. An in-place restore does not apply here, since it runshelm upgradewith the release of the backup, whose Istio install hook then runs over Istio 1.31.
Changes from 5.2.2#
These changes ship in 5.2.2 and 5.3.0. They apply to an upgrade from 5.1.x to 5.2.2 or later, from 5.2.0 or 5.2.1 to a later 5.2 release, and from 5.2.0 or 5.2.1 to 5.3.
OnlineFS drops records that arrive on another topic#
To address a vulnerability, OnlineFS writes a record only if it arrived on the online topic of the feature group that its headers name. A record that fails the check is dropped, logged by OnlineFS (Record dropped, it did not arrive on the online topic of the feature group in its headers) and counted in onlinefs_topic_mismatch_total, on the Topics row of the OnlineFS dashboard.
- Producers that hold on to an old topic. After a project's topic changes, or a feature group moves from the project topic to a topic of its own, the records that producers keep writing to the old topic are dropped until those producers restart. Producers that hold on to their topic include the inference logger of model deployments, streaming jobs and long-lived sessions.
- Records that reach OnlineFS on another topic name. A Kafka where records reach OnlineFS on a topic other than the one Hopsworks resolves for their feature group, such as a multi-region secondary that reads topics renamed or prefixed by mirroring, needs
onlinefs.configmapOverrides.onlinefssite.service.verifyFeatureGroupTopic: false. With the check off, OnlineFS logs a warning at startup and keeps the behaviour of 5.2.1, without the fix. - Projects that use another project's topic. Creating a project, changing its topic, creating an online feature group or changing its topic, or creating a Kafka topic fails with
TOPIC_IN_USE_BY_ANOTHER_PROJECT(190026) when the topic is the online topic of another project, the topic of another project's feature group, or a topic another project owns. Because of a security fix, give each project and online feature group that already uses another project's topic a topic of its own after the upgrade, then restart its producers as above; the upgrade leaves such a topic in place, and updates that keep it still succeed. A project whose own topic is taken that way cannot create online feature groups until it gets a topic of its own, while the project that created the topic, or whose default<project>_onlinefstopic it is, keeps using it. With an external Kafka (enable_bring_your_own_kafka), where Hopsworks owns no topics, the same security fix needs each project to have topics of its own there, and Kafka credentials that write only to them. From 5.3 a topic on an external Kafka is refused only when another project owns it because Hopsworks created it for that project, as described in Bring your own Kafka. - After the upgrade, check that the topic mismatch panels stay at zero. A drop means one of the cases above: its insert never completes its online ingestion, and the project's
get_failuresdoes not list the dropped records.
Schema migrations run on the DDL SQL node#
The hopsworks and HopsFS Flyway migrations and the Hive schematool connect to the DDL SQL node (mysqlddl, rondb.rondb.meta.ddlMySQLd), as the Airflow migrations already do. A migration no longer waits for application queries on the other SQL nodes that hold a related table, such as the statistics cleaner's query during migration V109. On a table that an application query holds itself, each statement waits up to ndb_schema_dist_lock_wait_timeout (30 seconds) and then completes with warning 1296 (Distribution of ALTER TABLE ... failed). The SQL node running that query keeps the old table definition: the query's transaction gets error 1412 (Table definition has changed) on its next statement on the table, and the first statement on the table after the transaction ends loads the new definition. The migration pods wait for the rollout of the DDL SQL node before they start. 5.1.3 carries the same change, so it also applies to an upgrade to 5.1.3.
The DDL SQL node is on by default. A cluster that disables it (rondb.rondb.meta.ddlMySQLd.enabled: false) points the migrations back at the application SQL nodes:
hopsworks:
dependencies:
mysql:
ddlConsulServiceName: mysql
hopsfs:
dependencies:
mysql:
ddlConsulServiceName: mysql
hive:
metastore:
dependencies:
mysql:
ddlConsulServiceName: mysql
The Airflow ddlConsulServiceName cannot point at mysql, because its migrations need a single SQL node, so a cluster without the DDL SQL node runs with global._hopsworks.airflow.enabled: false.
Changes from 5.2.3#
These changes ship in 5.2.3. They apply to an upgrade from 5.1.x to 5.2.3 or later, and from 5.2.0, 5.2.1 or 5.2.2 to a later 5.2 release.
Envoy image of Python apps#
Python apps pull their Envoy sidecar from global._hopsworks.imageRegistry, as <registry>/envoyproxy/envoy:<tag> with the tag in hopsworks.variables.python_app_envoy_image_version (v1.38.0); before, the chart did not set the backend's python_app_envoy_image, so clusters pulled envoyproxy/envoy:v1.38.0 from Docker Hub. The default registry, docker.hops.works, has the image. A cluster whose global._hopsworks.imageRegistry is its own registry mirrors envoyproxy/envoy:v1.38.0 there, at the path envoyproxy/envoy, before the upgrade, or its Python apps cannot start; the chart's vendor_images.sh does not list the image. The chart now writes python_app_envoy_image on every upgrade, replacing a value set by hand, so a cluster that pointed it at another image mirrors that image as above.
Changes from 5.2.5#
These changes ship in 5.2.5. They apply to an upgrade from 5.1.x to 5.2.5 or later, and from 5.2.4 or earlier to a later 5.2 release.
Istio 1.30#
Istio moves from 1.29 to 1.30. Istio 1.30 supports Kubernetes 1.32 to 1.36, and Istio supports the 1.30 release until about December 2026; 5.3 moves to Istio 1.31.
- Time. Each wait in the Istio install hook, the install and then istiod and each gateway rolling out, is bounded by
kserve.istio.istioctl.readinessTimeout(15 minutes), so allow at least an hour for the hook inhelm upgrade --timeout. - Model deployments. The Istio ingress gateway and the cluster-local gateway restart once. Each rolls out its new pod before it stops the old one.
- Your own image registry. Mirror
istio/pilotandistio/proxyv2at1.30.8before the upgrade. The chart'svendor_images.shlists both. - Sidecars you injected yourself. The chart injects no Istio sidecar. A workload you gave an Istio sidecar keeps its 1.29 proxy until it restarts, so restart it after the upgrade.
- If the hook fails. Fix the cause and run
helm upgradeagain. The hook stops with an error, and changes nothing, when istiod or the gateways run a release other than 1.29 or 1.30. - Going back to an earlier 5.2 release. A
helm rollbackdoes not run the Istio install hook, so Istio stays on 1.30. Do not runhelm upgrade, or an ArgoCD sync, with a release older than 5.2.5 afterwards: its hook installs Istio 1.29 over 1.30, a downgrade Istio does not support. To go back entirely, restore the backup on a new cluster, as described in New Cluster Restore. An in-place restore does not apply here, since it runshelm upgradewith the release of the backup, whose Istio install hook installs Istio 1.29 over 1.30.