Package com.logicalclocks.hsfs.spark.util

Class StorageConnectorUtils

java.lang.Object

com.logicalclocks.hsfs.spark.util.StorageConnectorUtils

public class StorageConnectorUtils
extends Object

Constructor Summary

Constructors

Constructor	Description
StorageConnectorUtils()

Method Summary

Modifier and Type	Method	Description
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.AdlsConnector connector, DataSource dataSource, String dataFormat, Map<String,String> options)	Reads path into a spark dataframe using the AdlsConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.BigqueryConnector connector, DataSource dataSource, Map<String,String> options)	Reads a query or a path into a spark dataframe using the sBigqueryConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.GcsConnector connector, DataSource dataSource, String dataFormat, Map<String,String> options)	Reads a path into a spark dataframe using the GcsConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.GlueConnector connector, DataSource dataSource, String dataFormat, Map<String,String> options)	Reads a Glue catalog table into a spark dataframe using the GlueConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.HopsFsConnector connector, DataSource dataSource, String dataFormat, Map<String,String> options)	Reads path into a spark dataframe using the HopsFsConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.JdbcConnector connector, DataSource dataSource, Map<String,String> options)	Reads query into a spark dataframe using the JdbcConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.RedshiftConnector connector, DataSource dataSource, Map<String,String> options)	Reads query into a spark dataframe using the RedshiftConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.S3Connector connector, DataSource dataSource, String dataFormat, Map<String,String> options)	Reads path into a spark dataframe using the S3Connector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.SnowflakeConnector connector, DataSource dataSource, Map<String,String> options)	Reads query into a spark dataframe using the SnowflakeConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector.SqlConnector connector, DataSource dataSource, Map<String,String> options)	Reads a query into a spark dataframe using the SqlConnector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(StorageConnector connector, DataSource dataSource, String dataFormat, Map<String,String> options)	Reads a query or a path into a spark dataframe using the storage connector.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	readStream(StorageConnector.KafkaConnector connector, String topic, boolean topicPattern, String messageFormat, String schema, Map<String,String> options, boolean includeMetadata)	Reads stream into a spark dataframe using the kafka storage connector.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

StorageConnectorUtils

public StorageConnectorUtils()

Method Details

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.HopsFsConnector connector,
 DataSource dataSource,
 String dataFormat,
 Map<String,String> options)
                                                            throws FeatureStoreException,
IOException

Reads path into a spark dataframe using the HopsFsConnector.

Parameters:

connector - HopsFsConnector object.

dataSource - Data source object.

dataFormat - specify the file format to be read, e.g. `csv`, `parquet`.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.S3Connector connector,
 DataSource dataSource,
 String dataFormat,
 Map<String,String> options)
                                                            throws FeatureStoreException,
IOException

Reads path into a spark dataframe using the S3Connector.

Parameters:

connector - S3Connector object.

dataSource - Data source object.

dataFormat - specify the file format to be read, e.g. `csv`, `parquet`.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.RedshiftConnector connector,
 DataSource dataSource,
 Map<String,String> options)
                                                            throws FeatureStoreException,
IOException

Reads query into a spark dataframe using the RedshiftConnector.

Parameters:

connector - Storage connector object.

dataSource - Data source object.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.AdlsConnector connector,
 DataSource dataSource,
 String dataFormat,
 Map<String,String> options)
                                                            throws FeatureStoreException,
IOException

Reads path into a spark dataframe using the AdlsConnector.

Parameters:

connector - AdlsConnector object.

dataSource - Data source object.

dataFormat - specify the file format to be read, e.g. `csv`, `parquet`.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.SnowflakeConnector connector,
 DataSource dataSource,
 Map<String,String> options)
                                                            throws FeatureStoreException,
IOException

Reads query into a spark dataframe using the SnowflakeConnector.

Parameters:

connector - SnowflakeConnector object.

dataSource - Data source object.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.JdbcConnector connector,
 DataSource dataSource,
 Map<String,String> options)
                                                            throws FeatureStoreException,
IOException

Reads query into a spark dataframe using the JdbcConnector.

Parameters:

connector - JdbcConnector object.

dataSource - Data source object.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.GcsConnector connector,
 DataSource dataSource,
 String dataFormat,
 Map<String,String> options)
                                                            throws FeatureStoreException,
IOException

Reads a path into a spark dataframe using the GcsConnector.

Parameters:

connector - GcsConnector object.

dataSource - Data source object.

dataFormat - Specify the file format to be read, e.g. `csv`, `parquet`.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.BigqueryConnector connector,
 DataSource dataSource,
 Map<String,String> options)
                                                            throws FeatureStoreException,
IOException

Reads a query or a path into a spark dataframe using the sBigqueryConnector.

Parameters:

connector - BigqueryConnector object.

dataSource - Data source object.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.GlueConnector connector,
 DataSource dataSource,
 String dataFormat,
 Map<String,String> options)
                                                            throws FeatureStoreException,
IOException

Reads a Glue catalog table into a spark dataframe using the GlueConnector.

Glue is a metadata catalog over data stored in S3, so reading mirrors the S3 connector: the underlying data lives at an S3 location (the data source path) and is read directly with the table's data format (e.g. `iceberg`, `delta`, `parquet`). The connector's AWS credentials are pushed into the Hadoop configuration the same way as for S3.

Parameters:

connector - GlueConnector object.

dataSource - Data source object.

dataFormat - specify the table format to be read, e.g. `iceberg`, `delta`, `parquet`.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.SqlConnector connector,
 DataSource dataSource,
 Map<String,String> options)
                                                            throws FeatureStoreException,
IOException

Reads a query into a spark dataframe using the SqlConnector.

Parameters:

connector - SqlConnector object.

dataSource - Data source object.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector connector,
 DataSource dataSource,
 String dataFormat,
 Map<String,String> options)
                                                            throws FeatureStoreException,
IOException

Reads a query or a path into a spark dataframe using the storage connector.

Parameters:

connector - Storage connector object.

dataSource - Data source object.

dataFormat - When reading from object stores such as S3, HopsFS and ADLS, specify the file format to be read, e.g. `csv`, `parquet`.

options - Any additional key/value options to be passed to the connector.

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

readStream

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> readStream(StorageConnector.KafkaConnector connector,
 String topic,
 boolean topicPattern,
 String messageFormat,
 String schema,
 Map<String,String> options,
 boolean includeMetadata)
                                                                  throws FeatureStoreException,
IOException

Reads stream into a spark dataframe using the kafka storage connector.

Parameters:

connector - Storage connector object.

topic - name of the topic.

topicPattern - if provided will subscribe topics that match provided pattern.

messageFormat - format of the message. "avro" or "json".

schema - schema of the message

options - Any additional key/value options to be passed to the connector.

includeMetadata - whether to include metadata of the topic in the dataframe, such as "key", "topic", "partition", offset", "timestamp", "timestampType", "value.*".

Returns:

Spark dataframe.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.