com.logicalclocks.hsfs.spark.engine.hudi

Class DeltaStreamerKafkaSource

java.lang.Object

org.apache.hudi.utilities.sources.Source<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>

org.apache.hudi.utilities.sources.AvroSource

com.logicalclocks.hsfs.spark.engine.hudi.DeltaStreamerKafkaSource

All Implemented Interfaces:

Serializable, org.apache.hudi.utilities.callback.SourceCommitCallback

public class DeltaStreamerKafkaSource
extends org.apache.hudi.utilities.sources.AvroSource

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.utilities.sources.Source

org.apache.hudi.utilities.sources.Source.SourceType

Field Summary

Fields inherited from class org.apache.hudi.utilities.sources.Source

props, sparkContext, sparkSession

Constructor Summary

Constructors

Constructor and Description
DeltaStreamerKafkaSource(org.apache.hudi.common.config.TypedProperties props, org.apache.spark.api.java.JavaSparkContext sparkContext, org.apache.spark.sql.SparkSession sparkSession, org.apache.hudi.utilities.schema.SchemaProvider schemaProvider)

Method Summary

Modifier and Type	Method and Description
protected org.apache.hudi.utilities.sources.InputBatch<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>	fetchNewData(org.apache.hudi.common.util.Option<String> lastCheckpointStr, long sourceLimit)
void	onCommit(String lastCkptStr)

Methods inherited from class org.apache.hudi.utilities.sources.Source

fetchNext, getSourceType, getSparkSession

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DeltaStreamerKafkaSource

public DeltaStreamerKafkaSource(org.apache.hudi.common.config.TypedProperties props,
                                org.apache.spark.api.java.JavaSparkContext sparkContext,
                                org.apache.spark.sql.SparkSession sparkSession,
                                org.apache.hudi.utilities.schema.SchemaProvider schemaProvider)

Method Detail

fetchNewData

protected org.apache.hudi.utilities.sources.InputBatch<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>> fetchNewData(org.apache.hudi.common.util.Option<String> lastCheckpointStr,
                                                                                                                                              long sourceLimit)

Specified by:

fetchNewData in class org.apache.hudi.utilities.sources.Source<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>

onCommit

public void onCommit(String lastCkptStr)