com.logicalclocks.hsfs.spark.util

Class StorageConnectorUtils

java.lang.Object

com.logicalclocks.hsfs.spark.util.StorageConnectorUtils

public class StorageConnectorUtils
extends Object

Constructor Summary

Constructors

Constructor and Description
StorageConnectorUtils()

Method Summary

Modifier and Type	Method and Description
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.AdlsConnector connector, DataSource dataSource, String dataFormat, Map<String,String> options) Reads path into a spark dataframe using the AdlsConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.BigqueryConnector connector, DataSource dataSource, Map<String,String> options) Reads a query or a path into a spark dataframe using the sBigqueryConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.GcsConnector connector, DataSource dataSource, String dataFormat, Map<String,String> options) Reads a path into a spark dataframe using the GcsConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.HopsFsConnector connector, DataSource dataSource, String dataFormat, Map<String,String> options) Reads path into a spark dataframe using the HopsFsConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.JdbcConnector connector, DataSource dataSource, Map<String,String> options) Reads query into a spark dataframe using the JdbcConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.RdsConnector connector, DataSource dataSource, Map<String,String> options) Reads a query or a path into a spark dataframe using the sBigqueryConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.RedshiftConnector connector, DataSource dataSource, Map<String,String> options) Reads query into a spark dataframe using the RedshiftConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.S3Connector connector, DataSource dataSource, String dataFormat, Map<String,String> options) Reads path into a spark dataframe using the S3Connector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.SnowflakeConnector connector, DataSource dataSource, Map<String,String> options) Reads query into a spark dataframe using the SnowflakeConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector connector, DataSource dataSource, String dataFormat, Map<String,String> options) Reads a query or a path into a spark dataframe using the storage connector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	readStream(StorageConnector.KafkaConnector connector, String topic, boolean topicPattern, String messageFormat, String schema, Map<String,String> options, boolean includeMetadata) Reads stream into a spark dataframe using the kafka storage connector.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StorageConnectorUtils

public StorageConnectorUtils()

Method Detail

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.HopsFsConnector connector,
                                                                   DataSource dataSource,
                                                                   String dataFormat,
                                                                   Map<String,String> options)
                                                            throws FeatureStoreException,
                                                                   IOException

Reads path into a spark dataframe using the HopsFsConnector.

Parameters:

connector - HopsFsConnector object.

dataSource - Data source object.

dataFormat - specify the file format to be read, e.g. `csv`, `parquet`.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.S3Connector connector,
                                                                   DataSource dataSource,
                                                                   String dataFormat,
                                                                   Map<String,String> options)
                                                            throws FeatureStoreException,
                                                                   IOException

Reads path into a spark dataframe using the S3Connector.

Parameters:

connector - S3Connector object.

dataSource - Data source object.

dataFormat - specify the file format to be read, e.g. `csv`, `parquet`.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.RedshiftConnector connector,
                                                                   DataSource dataSource,
                                                                   Map<String,String> options)
                                                            throws FeatureStoreException,
                                                                   IOException

Reads query into a spark dataframe using the RedshiftConnector.

Parameters:

connector - Storage connector object.

dataSource - Data source object.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.AdlsConnector connector,
                                                                   DataSource dataSource,
                                                                   String dataFormat,
                                                                   Map<String,String> options)
                                                            throws FeatureStoreException,
                                                                   IOException

Reads path into a spark dataframe using the AdlsConnector.

Parameters:

connector - AdlsConnector object.

dataSource - Data source object.

dataFormat - specify the file format to be read, e.g. `csv`, `parquet`.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.SnowflakeConnector connector,
                                                                   DataSource dataSource,
                                                                   Map<String,String> options)
                                                            throws FeatureStoreException,
                                                                   IOException

Reads query into a spark dataframe using the SnowflakeConnector.

Parameters:

connector - SnowflakeConnector object.

dataSource - Data source object.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.JdbcConnector connector,
                                                                   DataSource dataSource,
                                                                   Map<String,String> options)
                                                            throws FeatureStoreException,
                                                                   IOException

Reads query into a spark dataframe using the JdbcConnector.

Parameters:

connector - JdbcConnector object.

dataSource - Data source object.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.GcsConnector connector,
                                                                   DataSource dataSource,
                                                                   String dataFormat,
                                                                   Map<String,String> options)
                                                            throws FeatureStoreException,
                                                                   IOException

Reads a path into a spark dataframe using the GcsConnector.

Parameters:

connector - GcsConnector object.

dataSource - Data source object.

dataFormat - Specify the file format to be read, e.g. `csv`, `parquet`.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.BigqueryConnector connector,
                                                                   DataSource dataSource,
                                                                   Map<String,String> options)
                                                            throws FeatureStoreException,
                                                                   IOException

Reads a query or a path into a spark dataframe using the sBigqueryConnector.

Parameters:

connector - BigqueryConnector object.

dataSource - Data source object.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.RdsConnector connector,
                                                                   DataSource dataSource,
                                                                   Map<String,String> options)
                                                            throws FeatureStoreException,
                                                                   IOException

Reads a query or a path into a spark dataframe using the sBigqueryConnector.

Parameters:

connector - RdsConnector object.

dataSource - Data source object.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector connector,
                                                                   DataSource dataSource,
                                                                   String dataFormat,
                                                                   Map<String,String> options)
                                                            throws FeatureStoreException,
                                                                   IOException

Reads a query or a path into a spark dataframe using the storage connector.

Parameters:

connector - Storage connector object.

dataSource - Data source object.

dataFormat - When reading from object stores such as S3, HopsFS and ADLS, specify the file format to be read, e.g. `csv`, `parquet`.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

readStream

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> readStream(StorageConnector.KafkaConnector connector,
                                                                         String topic,
                                                                         boolean topicPattern,
                                                                         String messageFormat,
                                                                         String schema,
                                                                         Map<String,String> options,
                                                                         boolean includeMetadata)
                                                                  throws FeatureStoreException,
                                                                         IOException

Reads stream into a spark dataframe using the kafka storage connector.

Parameters:

connector - Storage connector object.

topic - name of the topic.

topicPattern - if provided will subscribe topics that match provided pattern.

messageFormat - format of the message. "avro" or "json".

schema - schema of the message

options - Any additional key/value options to be passed to the connector.

includeMetadata - whether to include metadata of the topic in the dataframe, such as "key", "topic", "partition", offset", "timestamp", "timestampType", "value.*".

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.