public class StorageConnectorUtils extends Object
Constructor and Description |
---|
StorageConnectorUtils() |
Modifier and Type | Method and Description |
---|---|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
read(StorageConnector.AdlsConnector connector,
DataSource dataSource,
String dataFormat,
Map<String,String> options)
Reads path into a spark dataframe using the AdlsConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
read(StorageConnector.BigqueryConnector connector,
DataSource dataSource,
Map<String,String> options)
Reads a query or a path into a spark dataframe using the sBigqueryConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
read(StorageConnector.GcsConnector connector,
DataSource dataSource,
String dataFormat,
Map<String,String> options)
Reads a path into a spark dataframe using the GcsConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
read(StorageConnector.HopsFsConnector connector,
DataSource dataSource,
String dataFormat,
Map<String,String> options)
Reads path into a spark dataframe using the HopsFsConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
read(StorageConnector.JdbcConnector connector,
DataSource dataSource,
Map<String,String> options)
Reads query into a spark dataframe using the JdbcConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
read(StorageConnector.RdsConnector connector,
DataSource dataSource,
Map<String,String> options)
Reads a query or a path into a spark dataframe using the sBigqueryConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
read(StorageConnector.RedshiftConnector connector,
DataSource dataSource,
Map<String,String> options)
Reads query into a spark dataframe using the RedshiftConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
read(StorageConnector.S3Connector connector,
DataSource dataSource,
String dataFormat,
Map<String,String> options)
Reads path into a spark dataframe using the S3Connector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
read(StorageConnector.SnowflakeConnector connector,
DataSource dataSource,
Map<String,String> options)
Reads query into a spark dataframe using the SnowflakeConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
read(StorageConnector connector,
DataSource dataSource,
String dataFormat,
Map<String,String> options)
Reads a query or a path into a spark dataframe using the storage connector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
readStream(StorageConnector.KafkaConnector connector,
String topic,
boolean topicPattern,
String messageFormat,
String schema,
Map<String,String> options,
boolean includeMetadata)
Reads stream into a spark dataframe using the kafka storage connector.
|
public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.HopsFsConnector connector, DataSource dataSource, String dataFormat, Map<String,String> options) throws FeatureStoreException, IOException
connector
- HopsFsConnector object.dataSource
- Data source object.dataFormat
- specify the file format to be read, e.g. `csv`, `parquet`.options
- Any additional key/value options to be passed to the connector.FeatureStoreException
- If unable to retrieve StorageConnector from the feature store.IOException
- Generic IO exception.public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.S3Connector connector, DataSource dataSource, String dataFormat, Map<String,String> options) throws FeatureStoreException, IOException
connector
- S3Connector object.dataSource
- Data source object.dataFormat
- specify the file format to be read, e.g. `csv`, `parquet`.options
- Any additional key/value options to be passed to the connector.FeatureStoreException
- If unable to retrieve StorageConnector from the feature store.IOException
- Generic IO exception.public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.RedshiftConnector connector, DataSource dataSource, Map<String,String> options) throws FeatureStoreException, IOException
connector
- Storage connector object.dataSource
- Data source object.options
- Any additional key/value options to be passed to the connector.FeatureStoreException
- If unable to retrieve StorageConnector from the feature store.IOException
- Generic IO exception.public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.AdlsConnector connector, DataSource dataSource, String dataFormat, Map<String,String> options) throws FeatureStoreException, IOException
connector
- AdlsConnector object.dataSource
- Data source object.dataFormat
- specify the file format to be read, e.g. `csv`, `parquet`.options
- Any additional key/value options to be passed to the connector.FeatureStoreException
- If unable to retrieve StorageConnector from the feature store.IOException
- Generic IO exception.public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.SnowflakeConnector connector, DataSource dataSource, Map<String,String> options) throws FeatureStoreException, IOException
connector
- SnowflakeConnector object.dataSource
- Data source object.options
- Any additional key/value options to be passed to the connector.FeatureStoreException
- If unable to retrieve StorageConnector from the feature store.IOException
- Generic IO exception.public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.JdbcConnector connector, DataSource dataSource, Map<String,String> options) throws FeatureStoreException, IOException
connector
- JdbcConnector object.dataSource
- Data source object.options
- Any additional key/value options to be passed to the connector.FeatureStoreException
- If unable to retrieve StorageConnector from the feature store.IOException
- Generic IO exception.public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.GcsConnector connector, DataSource dataSource, String dataFormat, Map<String,String> options) throws FeatureStoreException, IOException
connector
- GcsConnector object.dataSource
- Data source object.dataFormat
- Specify the file format to be read, e.g. `csv`, `parquet`.options
- Any additional key/value options to be passed to the connector.FeatureStoreException
- If unable to retrieve StorageConnector from the feature store.IOException
- Generic IO exception.public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.BigqueryConnector connector, DataSource dataSource, Map<String,String> options) throws FeatureStoreException, IOException
connector
- BigqueryConnector object.dataSource
- Data source object.options
- Any additional key/value options to be passed to the connector.FeatureStoreException
- If unable to retrieve StorageConnector from the feature store.IOException
- Generic IO exception.public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector.RdsConnector connector, DataSource dataSource, Map<String,String> options) throws FeatureStoreException, IOException
connector
- RdsConnector object.dataSource
- Data source object.options
- Any additional key/value options to be passed to the connector.FeatureStoreException
- If unable to retrieve StorageConnector from the feature store.IOException
- Generic IO exception.public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(StorageConnector connector, DataSource dataSource, String dataFormat, Map<String,String> options) throws FeatureStoreException, IOException
connector
- Storage connector object.dataSource
- Data source object.dataFormat
- When reading from object stores such as S3, HopsFS and ADLS, specify the file format to be read,
e.g. `csv`, `parquet`.options
- Any additional key/value options to be passed to the connector.FeatureStoreException
- If unable to retrieve StorageConnector from the feature store.IOException
- Generic IO exception.public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> readStream(StorageConnector.KafkaConnector connector, String topic, boolean topicPattern, String messageFormat, String schema, Map<String,String> options, boolean includeMetadata) throws FeatureStoreException, IOException
connector
- Storage connector object.topic
- name of the topic.topicPattern
- if provided will subscribe topics that match provided pattern.messageFormat
- format of the message. "avro" or "json".schema
- schema of the messageoptions
- Any additional key/value options to be passed to the connector.includeMetadata
- whether to include metadata of the topic in the dataframe, such as "key", "topic",
"partition", offset", "timestamp", "timestampType", "value.*".FeatureStoreException
- If unable to retrieve StorageConnector from the feature store.IOException
- Generic IO exception.Copyright © 2025. All rights reserved.