Package | Description |
---|---|
com.logicalclocks.hsfs | |
com.logicalclocks.hsfs.spark.util |
Modifier and Type | Field and Description |
---|---|
protected DataSource |
FeatureGroupBase.dataSource |
Modifier and Type | Method and Description |
---|---|
abstract Map<String,String> |
StorageConnector.sparkOptions(DataSource dataSource) |
Map<String,String> |
StorageConnector.HopsFsConnector.sparkOptions(DataSource dataSource) |
Map<String,String> |
StorageConnector.S3Connector.sparkOptions(DataSource dataSource) |
Map<String,String> |
StorageConnector.RedshiftConnector.sparkOptions(DataSource dataSource) |
Map<String,String> |
StorageConnector.AdlsConnector.sparkOptions(DataSource dataSource) |
Map<String,String> |
StorageConnector.SnowflakeConnector.sparkOptions(DataSource dataSource) |
Map<String,String> |
StorageConnector.JdbcConnector.sparkOptions(DataSource dataSource) |
Map<String,String> |
StorageConnector.KafkaConnector.sparkOptions(DataSource dataSource) |
Map<String,String> |
StorageConnector.GcsConnector.sparkOptions(DataSource dataSource) |
Map<String,String> |
StorageConnector.BigqueryConnector.sparkOptions(DataSource dataSource)
Set spark options specific to BigQuery.
|
Map<String,String> |
StorageConnector.RdsConnector.sparkOptions(DataSource dataSource)
Set spark options specific to Rds.
|
Modifier and Type | Method and Description |
---|---|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
StorageConnectorUtils.read(StorageConnector.AdlsConnector connector,
DataSource dataSource,
String dataFormat,
Map<String,String> options)
Reads path into a spark dataframe using the AdlsConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
StorageConnectorUtils.read(StorageConnector.BigqueryConnector connector,
DataSource dataSource,
Map<String,String> options)
Reads a query or a path into a spark dataframe using the sBigqueryConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
StorageConnectorUtils.read(StorageConnector.GcsConnector connector,
DataSource dataSource,
String dataFormat,
Map<String,String> options)
Reads a path into a spark dataframe using the GcsConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
StorageConnectorUtils.read(StorageConnector.HopsFsConnector connector,
DataSource dataSource,
String dataFormat,
Map<String,String> options)
Reads path into a spark dataframe using the HopsFsConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
StorageConnectorUtils.read(StorageConnector.JdbcConnector connector,
DataSource dataSource,
Map<String,String> options)
Reads query into a spark dataframe using the JdbcConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
StorageConnectorUtils.read(StorageConnector.RdsConnector connector,
DataSource dataSource,
Map<String,String> options)
Reads a query or a path into a spark dataframe using the sBigqueryConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
StorageConnectorUtils.read(StorageConnector.RedshiftConnector connector,
DataSource dataSource,
Map<String,String> options)
Reads query into a spark dataframe using the RedshiftConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
StorageConnectorUtils.read(StorageConnector.S3Connector connector,
DataSource dataSource,
String dataFormat,
Map<String,String> options)
Reads path into a spark dataframe using the S3Connector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
StorageConnectorUtils.read(StorageConnector.SnowflakeConnector connector,
DataSource dataSource,
Map<String,String> options)
Reads query into a spark dataframe using the SnowflakeConnector.
|
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
StorageConnectorUtils.read(StorageConnector connector,
DataSource dataSource,
String dataFormat,
Map<String,String> options)
Reads a query or a path into a spark dataframe using the storage connector.
|
Copyright © 2025. All rights reserved.