Skip to content

KafkaTopic API#

Handle#

[source]

get_kafka_api#

Project.get_kafka_api()

Get the kafka api for the project.

Returns

KafkaApi: The Kafka Api handle

Configuration#

[source]

get_default_config#

KafkaApi.get_default_config(internal_kafka=None)

Get the configuration to set up a Producer or Consumer for a Kafka broker using confluent-kafka.

import hopsworks

project = hopsworks.login()

kafka_api = project.get_kafka_api()

kafka_conf = kafka_api.get_default_config()

from confluent_kafka import Producer

producer = Producer(kafka_conf)
Returns

dict: The kafka configuration

Raises

  • hopsworks.client.exceptions.RestAPIError: If the backend encounters an error when handling the request

Creation#

[source]

create_topic#

KafkaApi.create_topic(name, schema, schema_version, replicas=1, partitions=1)

Create a new kafka topic.

import hopsworks

project = hopsworks.login()

kafka_api = project.get_kafka_api()

kafka_topic = kafka_api.create_topic("my_topic", "my_schema", 1)
Arguments

  • name str: name of the topic
  • schema str: subject name of the schema
  • schema_version int: version of the schema
  • replicas int: replication factor for the topic
  • partitions int: partitions for the topic

Returns

KafkaTopic: The KafkaTopic object

Raises

  • hopsworks.client.exceptions.RestAPIError: If the backend encounters an error when handling the request

Retrieval#

[source]

get_topic#

KafkaApi.get_topic(name)

Get kafka topic by name.

Arguments

  • name str: name of the topic

Returns

KafkaTopic: The KafkaTopic object or None if not found

Raises

  • hopsworks.client.exceptions.RestAPIError: If the backend encounters an error when handling the request

[source]

get_topics#

KafkaApi.get_topics()

Get all kafka topics.

Returns

List[KafkaTopic]: List of KafkaTopic objects

Raises

  • hopsworks.client.exceptions.RestAPIError: If the backend encounters an error when handling the request

Properties#

[source]

name#

Name of the Kafka topic.

[source]

num_partitions#

Number of partitions of the Kafka topic.

[source]

num_replicas#

Number of replicas of the Kafka topic.

[source]

partitions#

Number of partitions of the Kafka topic.

[source]

replicas#

Number of replicas of the Kafka topic.

[source]

schema#

Schema for the topic

[source]

num_partitions#

Number of partitions of the Kafka topic.

Methods#

[source]

delete#

KafkaTopic.delete()

Delete the topic

Potentially dangerous operation

This operation deletes the topic.

Raises

  • hopsworks.client.exceptions.RestAPIError: If the backend encounters an error when handling the request

[source]

describe#

KafkaTopic.describe()