OnlineFS Configuration#
Introduction#
The Online Feature Store service (OnlineFS in short) consumes the feature store topics from Kafka, and writes the feature group rows to the online feature store (RonDB) and the embeddings to the vector database (OpenSearch). Administrators configure it in Hopsworks under Cluster Settings → OnlineFS.
The onlinefs Helm chart keeps deploying OnlineFS. Replicas, resources, scheduling, autoscaling and the disruption budget stay in the OnlineFS Helm values. Hopsworks keeps the settings in its database and changes only one annotation on the pod template of the OnlineFS Deployment, which the chart does not render, so a helm upgrade undoes neither and ArgoCD reports no drift.
Hopsworks stores four groups of settings:
- The Kafka connection.
- The topic subscription.
- The Kafka client properties of each Kafka client of OnlineFS.
- The service settings.
At startup, OnlineFS reads its Helm configuration, then fetches the saved settings from Hopsworks and applies them over it. A saved setting wins over the Helm values and over the environment variables the chart sets. A setting that was never saved keeps its Helm value, and with nothing saved OnlineFS runs with its Helm configuration. Hopsworks applies a change by restarting the OnlineFS pods.
Users who are not administrators see the status read-only in their project settings, without the revisions and the restart error.
Kafka connection#
The connection is a set of Kafka client properties. Left empty, which is the default, OnlineFS connects as its Helm configuration says: to the Kafka cluster bundled with Hopsworks, unless onlinefs.dependencies.kafka points it elsewhere.
A connection that is set must contain bootstrap.servers, and it replaces the Helm connection as a group. OnlineFS removes the connection keys from the Helm configuration of each of its Kafka clients and uses the entered ones in their place. The connection keys are bootstrap.servers, bootstrap.controllers, security.protocol, security.providers, and every key starting with ssl. or sasl.. None of the TLS settings of the chart carry over: the truststore and keystore of the Hopsworks certificates, their passwords, and the empty ssl.endpoint.identification.algorithm, which turns off host name verification. Add the ones the cluster needs to the connection.
The connection applies to every Kafka client of OnlineFS: the consumer that writes to RonDB, the consumer that writes to the vector database, the producer of feature group notifications, and the admin client that reports the consumer group metrics. Because they all follow one connection, the client properties of a single client cannot contain connection keys.
A connection to a cluster with SASL_SSL and SCRAM:
bootstrap.servers=b-1.kafka.example.com:9096,b-2.kafka.example.com:9096
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="onlinefs" password="<password>";
Without truststore settings, OnlineFS accepts broker certificates signed by a certificate authority in the default truststore of its JVM.
With an external Kafka cluster, enter the cluster that the kafka_connector data sources of the projects use.
Certificates#
Certificates and keys go into the connection either inline in PEM form, or as paths to files inside the OnlineFS pod. Hopsworks stores the connection as properties only, so a truststore or keystore file has to exist in the pod already.
In PEM form:
| Key | Value |
|---|---|
ssl.truststore.type | PEM |
ssl.truststore.certificates | The certificates of the certificate authorities that signed the broker certificates. |
ssl.keystore.type | PEM, for a cluster that authenticates clients by certificate. |
ssl.keystore.certificate.chain | The client certificate chain. |
ssl.keystore.key | The private key of the client, in PKCS#8 form. |
ssl.key.password | The password of the private key, when it is encrypted. |
A PEM value keeps its line breaks, from the -----BEGIN line to the -----END line. Line breaks are accepted only in the values of keys ending in .certificates, .certificate.chain or .key, which can be up to 5000 characters long. Every other value, in any of the settings on this page, is a single line of at most 1000 characters. Outside the Kafka connection, keys and values can use only Latin-1 characters, each of the four property maps can hold at most 5000 characters in total, and the topic pattern and the comma-separated topic list are each at most 1000 characters. The Kafka connection as a whole can hold up to 64 KiB, enough for a CA certificate, a client certificate and a key given inline.
As files, ssl.truststore.location and ssl.keystore.location point at a truststore and a keystore inside the OnlineFS pod, with their passwords in ssl.truststore.password, ssl.keystore.password and ssl.key.password. The chart mounts the Hopsworks certificates under /certs.
Credentials#
Hopsworks stores the connection encrypted in its database with the master encryption password, and re-encrypts it when that password changes. A first connection saved while a master encryption password change is running can stay encrypted with the old password. An OnlineFS pod that starts then cannot fetch its settings and does not consume, and the status shows CONNECTION_UNREADABLE.
The settings cannot be read either, so the page offers Reset to Helm configuration in place of the form. The reset clears the connection, and it can be entered again.
Hopsworks shows the value of every key whose name contains password, jaas or secret, or ends in .key, as ********. Saving with ******** left in place keeps the stored value, so a credential has to be entered again only to change it. Only OnlineFS receives the values unmasked, and neither Hopsworks nor OnlineFS logs them.
Subscription#
OnlineFS subscribes to its topics either by a pattern or by a list:
- A pattern is a Java regular expression, by default
.*_onlinefs. OnlineFS consumes every topic whose name it matches, including topics created later. - A topic list names the topics, separated by commas. Hopsworks trims the names and drops empty entries and duplicates. A name can contain letters, digits,
.,_and-, and be up to 249 characters long.
A topic created later is not consumed until it is added to the list, so its data does not reach the online feature store until then. Hopsworks creates the topics of new feature groups and projects, so subscribe by pattern unless every topic is known in advance. To consume project topics that do not end in _onlinefs, widen the pattern, for example to .*_onlinefs|fraud_transactions. The ingestion topic guide describes which topic a feature group writes to.
Left empty, OnlineFS keeps the subscription of its Helm values, under onlinefs.configmapOverrides.onlinefssite.kafkaConsumer. A saved subscription replaces both Helm values, topicPattern and topicList, because a non-empty topic list in the Helm values would otherwise take precedence over a saved pattern.
Kafka client properties#
Three sets of Kafka client properties tune one client each:
- The consumer properties apply to the consumer that writes feature group rows to RonDB, in the consumer group
onlinefs_rondb. - The VectorDB consumer properties apply to the consumer that writes embeddings to the vector database, in the consumer group
onlinefs_vectordb. - The producer properties apply to the producer of feature group notifications.
Each set is applied last, over the properties the chart sets for that client and over the connection. For example, max.poll.records=500 in the consumer properties replaces the 1000 the chart sets. The properties the chart sets are in onlinefs-kafka.properties, onlinefs-kafka-vector-db.properties and producer.properties of the OnlineFS ConfigMap.
A set that contains a connection key is refused when saving, because every client follows the one Kafka connection.
Service settings#
Service settings are keys of onlinefs-site.xml, written as section.key. Only the keys below are accepted, with a value of the listed type, and every integer must be 0 or more. service.threadNumber and service.reportingQueueSizePerThread must be at least 1, because OnlineFS sizes its reporting thread pool and queues from them and cannot start with 0. rondb.batchSize and opensearch.batchSize must be at least 1, because OnlineFS splits the rows it writes into batches of that size. service.ronDbThreadNumber and service.vectorDbThreadNumber cannot both be 0, because OnlineFS would then consume nothing. The default is the value the chart sets, or the default of OnlineFS where the chart sets none.
service
| Key | Type | Default | Description |
|---|---|---|---|
service.threadNumber | integer | 10 | Maximum number of threads that report ingestion to Hopsworks and send feature group notifications. |
service.ronDbThreadNumber | integer | 10 | Number of Kafka consumer threads that write to RonDB. |
service.vectorDbThreadNumber | integer | 10, or 0 without VectorDB ingestion | Number of Kafka consumer threads that write embeddings to the vector database. Set it above 0 only when onlinefs.deployment.vectordb.enabled is true, because only then does the chart provide the OpenSearch credentials. |
service.reportingQueueSizePerThread | integer | 100 | Queue size per reporting and notification thread. |
service.getSessionRetrySleepMs | integer | 100 | Milliseconds between attempts to get a RonDB session. |
service.pauseRetrySleepMs | integer | 5000 | Milliseconds between attempts while ingestion is paused on a failure that looks transient, such as a RonDB or Hopsworks upgrade. |
service.maxPauseMs | integer | 0 | Milliseconds after which OnlineFS gives up on a batch paused on such a failure, and drops its rows. 0 waits until the failure resolves. |
service.maxBlacklistSize | integer | 100 | Maximum number of entries in the lists of feature groups and projects whose records OnlineFS skips. |
service.maxFeatureGroupCacheSize | integer | 1000 | Maximum number of entries in the feature group metadata cache. |
service.maxFeatureStoreCacheSize | integer | 1000 | Maximum number of entries in the feature store metadata cache. |
service.maxFeatureViewCacheSize | integer | 1000 | Maximum number of entries in the feature view metadata cache. |
service.featureGroupCacheExpire | integer | 30 | Minutes after which a feature group cache entry expires. |
service.featureStoreCacheExpire | integer | 30 | Minutes after which a feature store cache entry expires. |
service.featureViewCacheExpire | integer | 10 | Minutes after which a feature view cache entry expires. |
service.verifyFeatureGroupTopic | boolean | true | Drop records that did not arrive on the online topic of the feature group their headers name. See OnlineFS drops records that arrive on another topic before turning it off. |
service.region | string | none | Region written into feature group notifications. |
rondb
| Key | Type | Default | Description |
|---|---|---|---|
rondb.batchSize | integer | 300 | Maximum number of rows per RonDB commit. |
rondb.maxRetries | integer | 3 | Retries of a failed RonDB commit. |
rondb.missingTableRetries | integer | 5 | Lookups of a table that RonDB reports as missing, before OnlineFS treats its feature group as deleted. |
rondb.maxCachedSessions | integer | 20 | Maximum number of ClusterJ sessions cached per connection. |
rondb.maxCachedInstances | integer | 1024 | Maximum number of ClusterJ instances cached. |
rondb.reconnectTimeout | integer | 5 | ClusterJ reconnect timeout, in seconds. |
rondb.maxTransactions | integer | 1024 | Maximum number of concurrent ClusterJ transactions. |
rondb.useSessionCache | boolean | false | Return ClusterJ sessions to a cache instead of closing them. |
rondb.useDynamicObjectCache | boolean | false | Return ClusterJ objects to a cache instead of releasing them. |
kafkaConsumer
| Key | Type | Default | Description |
|---|---|---|---|
kafkaConsumer.pollTimeoutMs | integer | 1000 | Kafka consumer poll timeout, in milliseconds. |
opensearch
| Key | Type | Default | Description |
|---|---|---|---|
opensearch.batchSize | integer | 1000 | Maximum number of embeddings per vector database commit. |
opensearch.maxRetries | integer | 10 | Retries of a failed vector database commit. |
Every other key is refused:
kafkaConsumer.topicPatternandkafkaConsumer.topicListare set through the subscription.hopsworks.*,kafka.propertiesFile*,metrics.port,rondb.connectionString,opensearch.host,opensearch.userNameandopensearch.passwordcome from the deployment: the endpoints, the credentials, the files the chart mounts, and the port its probes check.rondb.poolSizestays in the Helm values, because every pooled connection of every pod takes a RonDB API slot from the pool that the HopsFS NameNodes also use.
Applying changes#
Save, Restart and Reset to Helm configuration each restart OnlineFS:
Savestores the settings.Restartleaves the settings as they are.Reset to Helm configurationclears every saved setting, so OnlineFS returns to its Helm configuration.
Hopsworks restarts OnlineFS with a rollout of its Deployment, as kubectl rollout restart does. It writes the revision of the saved settings into the onlinefs.hopsworks.ai/settings-revision annotation of the pod template, and Kubernetes replaces the pods. Each new pod fetches the settings when it starts. A change saved during a rollout is applied by a further rollout once that one ends.
The rollout strategy of the Deployment decides the order in which pods are replaced:
- With a single replica the chart sets no strategy, so Kubernetes starts the new pod first and stops the old one once the new one is ready.
- With two or more replicas the chart stops a pod before it starts its replacement, so that a rollout completes on a node pool with no room for an extra pod. The other pods keep consuming meanwhile.
onlinefs.deployment.strategyin the Helm values replaces both, for example withmaxSurge: 1andmaxUnavailable: 0to start every new pod first. It applies to Helm upgrades as well.
A new pod started first needs room on the node pool, and free RonDB API slots for its connections. Kubernetes counts a pod as ready once its metrics port answers, so an old pod can stop shortly before its replacement consumes. Kafka keeps the records while pods are replaced, and OnlineFS continues from its stored offsets.
When Kubernetes reports that the rollout cannot progress, for example because a new pod cannot be scheduled or keeps crashing, the restart stops and the status shows the settings as FAILED, with the reason. Fix the cause and press Restart.
Status#
The status shows the state of the OnlineFS Deployment, its desired, ready, updated and available replicas, and whether the latest settings are applied.
| State | Meaning |
|---|---|
RUNNING | The Deployment runs its replicas and no rollout is in progress. |
UPDATING | A rollout is in progress. |
STOPPED | The Deployment has 0 replicas. |
FAILED | Kubernetes reports that the rollout cannot progress, with the reason. |
NOT_FOUND | No Deployment named by the onlinefs_deployment_name variable, by default onlinefs-deployment, exists in the namespace Hopsworks is installed in. |
UNKNOWN | Hopsworks does not run on Kubernetes. |
| Settings state | Meaning |
|---|---|
NONE | Nothing was ever saved, and OnlineFS runs with its Helm configuration. |
PENDING | Settings were saved or reset, or Restart was pressed, and the rollout has not completed yet. |
FAILED | The last restart stopped on an error, shown with the status. |
NOT_FETCHED | The pods were replaced, but none fetched the latest settings, because the OnlineFS image they run does not fetch settings from Hopsworks. |
CONNECTION_UNREADABLE | The saved Kafka connection does not decrypt with the current master encryption password, so an OnlineFS pod that starts cannot fetch its settings. It takes precedence over the other states. See Credentials. |
APPLIED | The pods were replaced and fetched the latest settings. |
APPLIED says nothing about whether OnlineFS reaches the Kafka cluster. The OnlineFS logs and the OnlineFS dashboard show that.
Administrators also see the revision of the saved settings, the revision the last completed restart applied, the revision OnlineFS fetched last, and the error of the last restart.
Consumer offsets#
OnlineFS stores its consumer offsets in the onlinefs.kafka_offsets table of the RonDB it writes to, per consumer group, topic and partition, with the ID Kafka gives the topic the offset was saved for. When a topic is deleted and created again under the same name, after a restore that rebuilt Kafka or by hand, Kafka gives it a new ID, and OnlineFS reads each partition of the new topic from its start instead of resuming at an offset of the earlier topic. A running OnlineFS sees the new topic at its next Kafka metadata refresh, every 60 seconds with the chart's metadata.max.age.ms. If a topic is deleted and created again between two refreshes, OnlineFS may keep its position on the new topic until it restarts, and then reads the new topic from its start.
OnlineFS keeps the offsets of a saved connection apart from those of the Helm connection, and starts new ones whenever the saved bootstrap.servers changes:
- With the Helm connection, OnlineFS uses the offsets it has always used, so an upgraded cluster resumes where it was.
- A saved connection whose
bootstrap.serversdiffers from the previous one, or a connection saved where there was none, starts a new set of offsets. OnlineFS then reads that cluster from the earliest records it retains, because the chart setsauto.offset.reset=earliest. This also holds for a cluster OnlineFS used before. - Changing any other key of the connection, such as the credentials, keeps the offsets.
- Resetting to the Helm configuration returns to the offsets of the Helm connection.
Hopsworks tells Kafka clusters apart by bootstrap.servers alone. A new broker list for the same cluster, or moving a connection from the Helm values into Cluster Settings, also makes OnlineFS read the retained records again and write them to the online feature store again. Plan such a change for a time when that load is acceptable. A change of the Helm connection itself, in the Helm values, keeps the offsets of the Helm connection, as in earlier releases.
Kafka offsets#
The Kafka offsets card lists the topics OnlineFS consumes: for each project with an online feature group, every topic its online feature groups write to, the project's online topic or one a feature group names itself. Each row shows the project, the topic and its number of partitions in Kafka. Read only marks a topic whose offsets cannot be changed, with the reason in its tooltip, as listed in Changing offsets. The search matches the project or the topic name.
A click on a topic opens its offsets, for each consumer group and partition:
Min offset: the first offset Kafka still has.Current offset: the offset OnlineFS resumes the partition from.Max offset: the end offset, where Kafka writes the next record. A current offset equal to it means OnlineFS has processed every record of the partition.Lag: the records OnlineFS has still to process, written or not.
Every partition Kafka has is listed. A current offset of - means OnlineFS reads the partition from its min offset: it saved no offset for it, or saved one for an earlier topic of the same name, as after a restore that recreated the topic, which is marked Earlier topic.
OnlineFS consumes each topic in two consumer groups: RONDB, which writes the online feature store, and VECTORDB, which writes embeddings to the vector database. With a saved Kafka connection the groups carry a number after an @, such as RONDB@3, which changes with the saved bootstrap.servers as described above. A consumer group marked Not read belongs to an earlier Kafka connection, and OnlineFS no longer reads its offsets. RONDB is always listed, and VECTORDB once it saved an offset for the topic. OnlineFS runs consumers of a group only while its thread number is above 0: with service.vectorDbThreadNumber at 0, the default without VectorDB ingestion, the VECTORDB offsets stay as they were.
The offsets are those in the RonDB of the region the Hopsworks backend runs in.
Changing offsets#
In a topic's offsets, an administrator can move partitions of one consumer group:
Set offsetmoves one selected partition to a given offset, from the first offset Kafka still has up to the end offset.Move to startmoves the selected partitions, or all of them, to the first offset Kafka still has, to process their records again.Move to endmoves them to the end offset, to skip every record written until then.
The confirmation lists each partition with its min, current and max offset and the new offset. Its button is red for a change that skips records: Move to end, or an offset above the current one. Hopsworks then sets the new offset as the partition's desired_offset in onlinefs.kafka_offsets, the start and the end as Kafka has them at that moment. The OnlineFS consumer that holds the partition looks for a new offset every 10 seconds, between batches, and moves the partition there with no restart: it saves the new offset, clears desired_offset, records the outcome, and resumes the partition from it. The page reads the topic's offsets again every 5 seconds while its browser tab is visible, and says when the next read is. Under its current offset, a partition shows Moving to <offset> until OnlineFS moved it, then Moved to <offset> by <time>, the end of the first read that showed the move, until the page is closed. After a move back, OnlineFS can catch up before that read, so the current offset may already be at the max offset again.
- Ingestion of other partitions and topics goes on.
- The
RONDBandVECTORDBgroups move separately. - Moving back processes the records again under the current rules: rows are written and deletes applied again, ingestion notifications repeat, and readers can see older values until OnlineFS catches up. Records Kafka no longer retains cannot be read again.
- Moving forward skips records, which never reach the online feature store.
- The offline feature store is not affected.
Setting another offset for a partition before OnlineFS moved it replaces the first, also when two administrators set one at the same time. OnlineFS clears a desired offset without moving the partition when the topic was deleted and created again since it was set, once OnlineFS has seen the new topic at its metadata refresh, or when Kafka no longer has the offset, as when it deleted older records meanwhile, and logs a warning. The page then shows Not moved to <offset> with the reason instead of Moved to <offset>. A partition no OnlineFS consumer holds, as in a group whose thread number is 0, moves when a consumer gets it. Hopsworks logs who set which offsets; OnlineFS logs each move.
A topic is read only, and its offsets cannot be changed, when:
- the project's online feature store is on a satellite, whose RonDB this cluster does not change;
- Kafka could not be asked, has no topic of this name, or gives the topic no ID.
Startup#
Each OnlineFS pod asks Hopsworks for the saved settings once at startup, before it consumes. Hopsworks answers only the OnlineFS service user, named by the onlinefs_service_user_email variable, by default onlinefs@hopsworks.ai, because the answer contains the Kafka credentials.
- When Hopsworks answers, OnlineFS applies the saved settings, or runs with its Helm configuration when nothing is saved.
- When Hopsworks is an older release without these settings, OnlineFS runs with its Helm configuration.
- When Hopsworks cannot be reached, answers with a server error, or refuses the API key, OnlineFS logs the error and asks again, after 1 second at first and doubling up to 60 seconds.
In the last case OnlineFS does not fall back to its Helm configuration, because that could make it consume from another Kafka cluster than the one saved, while waiting loses nothing: Kafka keeps the records. A pod that starts while Hopsworks is down therefore waits for Hopsworks, whether settings are saved or not. Running pods are not affected, since they ask only at startup.
Permissions on the Kafka cluster#
On a Kafka cluster with authorization, the principal of the connection needs:
| Resource | Operations | Needed for |
|---|---|---|
| The topics OnlineFS subscribes to | Describe, Read | Consuming the feature group rows. |
The consumer groups onlinefs_rondb and onlinefs_vectordb | Describe, Read | Joining the consumer groups and reporting their metrics. |
| The notification topics of feature groups | Write | Sending feature group notifications. |
A group.id in the consumer properties or the VectorDB consumer properties changes the name of the consumer group to grant. OnlineFS creates no topics. On an external Kafka cluster, Hopsworks creates them with the kafka_connector data source of each project, as described in Topics on the external Kafka cluster.
REST API#
Administrators can manage the settings with the REST API, also with an API key of scope ADMIN:
GET /hopsworks-api/api/admin/onlinefs/config
PUT /hopsworks-api/api/admin/onlinefs/config
DELETE /hopsworks-api/api/admin/onlinefs/config
POST /hopsworks-api/api/admin/onlinefs/restart
GET /hopsworks-api/api/admin/onlinefs/status
PUT saves the settings and DELETE resets them to the Helm configuration, and both return the resulting settings. POST on restart restarts OnlineFS with the settings unchanged and returns the status. All three restart OnlineFS as described in Applying changes.
{
"kafkaConnection": {
"bootstrap.servers": "b-1.kafka.example.com:9096",
"security.protocol": "SASL_SSL",
"sasl.mechanism": "SCRAM-SHA-512",
"sasl.jaas.config": "********"
},
"topicPattern": ".*_onlinefs",
"topicList": null,
"kafkaConsumerOverrides": {"max.poll.records": "500"},
"kafkaProducerOverrides": {},
"kafkaVectordbOverrides": {},
"siteOverrides": {"service.ronDbThreadNumber": "12"},
"revision": 7,
"updated": "2026-10-27T15:00:00.000Z"
}
- A
PUTreplaces every setting, so send the whole configuration as aGETreturns it. - An empty or absent
kafkaConnectionkeeps the Helm connection. - At most one of
topicPatternandtopicListis set, andtopicListis an array of topic names. With bothnull, OnlineFS keeps the subscription of its Helm values. - A value of
********keeps the stored value of that key. revisionandupdatedare read-only and ignored onPUT.
Every logged-in user can read the status of OnlineFS with GET /hopsworks-api/api/onlinefs/status, also with an API key of scope FEATURESTORE. It returns the state, the replica counts, the message, the settings state, and whether any setting is saved, without the revisions and the restart error.
Kafka offsets#
Administrators can read and change the Kafka offsets, also with an API key of scope ADMIN:
GET /hopsworks-api/api/admin/onlinefs/offsets?filter=<text>&offset=<n>&limit=<n>
GET /hopsworks-api/api/admin/onlinefs/offsets/projects/{projectId}/topics/{topic}
POST /hopsworks-api/api/admin/onlinefs/offsets/projects/{projectId}/topics/{topic}/changes
- The list returns a page of
items, by project and topic name, withcount, the number of topics the filter matches;limitis at most 100. - A topic returns its
topicId, itsreadOnlyReasonwhen its offsets cannot be changed, anditemswith an entry per consumer group and partition. A field left out is unknown, such as theoffsetof a partition OnlineFS saved none for. - Each entry of
itemshas theoffsetOnlineFS saved, thedesiredOffsetset that OnlineFS has not moved the partition to yet,beginningOffsetandendOffset, the offsets Kafka has,lag,activeConsumerGroupand astatus.lagis the number of records OnlineFS has still to process, written or not; it is set for the current topic, and for a partition with no saved offset, in a consumer group of the current connection. lastMoveOffsetis the last desired offset OnlineFS took up for the partition, andlastMoveOutcomewhat it did with it:MOVED, or, without moving the partition,EARLIER_TOPIC(the topic was created again since),OUT_OF_RANGE(Kafka no longer has the offset) orNO_TOPIC_ID(Kafka gives the topic no ID).
status | Meaning |
|---|---|
CURRENT_TOPIC | The offset was saved for the topic Kafka has under this name now, so OnlineFS resumes the partition from it. |
EARLIER_TOPIC | The offset was saved for an earlier topic of the same name, so OnlineFS reads the partition from its start. |
TOPIC_ID_UNKNOWN | The offset was saved without a topic ID, or Kafka gives its topics none, so which topic it belongs to is not known. |
TOPIC_MISSING | Kafka has no topic of this name. |
KAFKA_UNAVAILABLE | Hopsworks could not ask Kafka, so the offsets Kafka has and the lag are unknown. |
NOT_SAVED | OnlineFS saved no offset for the partition, so it reads it from its start. |
A POST sets the offset partitions of one consumer group resume from:
{
"consumerGroup": "RONDB",
"partitions": [3],
"target": "OFFSET",
"offset": 1981,
"topicId": "qIquf00TSxOofDURgAgSlg"
}
targetisOFFSET,EARLIESTorLATEST, andoffsetis set forOFFSETonly.EARLIESTandLATESTare the offsets Kafka has when the request is made.- An empty
partitionssets every partition the topic has. topicIdrepeats the one the topic returned, so the request is refused once the topic was deleted and created again since.- It answers
204 No Contentonce the offsets are set; OnlineFS moves the partitions as described in Changing offsets. Setting the same offsets again is safe.
| Status | Error code | When |
|---|---|---|
| 400 | 270378 | The request is invalid: no target, a consumer group OnlineFS does not read, an offset with EARLIEST or LATEST, a partition the topic does not have, or an offset outside the partition's range. |
| 404 | 270375 | No online feature group of the project writes to the topic. |
| 409 | 270376 | The topic is read only; the message gives the reason. |
| 409 | 270377 | The topic was deleted and created again since its offsets were read. |
| 503 | 270374 | The offsets cannot be read or set, as before OnlineFS has created onlinefs.kafka_offsets. When RonDB did not confirm the commit, the message says they may have been set: setting them again is safe. |