Storage Connector#
Storage connectors encapsulate all information needed for the execution engine to read and write to specific storage. This storage can be S3, a JDBC compliant database or the distributed filesystem HOPSFS.
Retrieval#
get_storage_connector#
FeatureStore.get_storage_connector(name)
Get a previously created storage connector from the feature store.
Storage connectors encapsulate all information needed for the execution engine to read and write to specific storage. This storage can be S3, a JDBC compliant database or the distributed filesystem HOPSFS.
If you want to connect to the online feature store, see the
get_online_storage_connector method to get the JDBC connector for the Online
Feature Store.
Getting a Storage Connector
sc = fs.get_storage_connector("demo_fs_meb10000_Training_Datasets")
td = fs.create_training_dataset(..., storage_connector=sc, ...)
Arguments
- name
str: Name of the storage connector to retrieve.
Returns
StorageConnector. Storage connector object.
get_online_storage_connector#
FeatureStore.get_online_storage_connector()
Get the storage connector for the Online Feature Store of the respective project's feature store.
The returned storage connector depends on the project that you are connected to.
Returns
StorageConnector. JDBC storage connector to the Online Feature Store.
Properties#
access_key#
Access key.
account#
Account of the Snowflake storage connector
account_name#
Account name of the ADLS storage connector
application_id#
Application ID of the ADLS storage connector
arguments#
Additional JDBC, REDSHIFT, or Snowflake arguments.
auto_create#
Database username for redshift cluster.
bucket#
Return the bucket for S3 connectors.
cluster_identifier#
Cluster identifier for redshift cluster.
connection_string#
JDBC connection string.
connector_type#
Type of the connector. S3, JDBC, REDSHIFT or HOPSFS.
container_name#
Container name of the ADLS storage connector
database#
Database of the Snowflake storage connector
database_driver#
Database endpoint for redshift cluster.
database_endpoint#
Database endpoint for redshift cluster.
database_group#
Database username for redshift cluster.
database_name#
Database name for redshift cluster.
database_password#
Database password for redshift cluster.
database_port#
Database port for redshift cluster.
database_user_name#
Database username for redshift cluster.
directory_id#
Directory ID of the ADLS storage connector
expiration#
Cluster temporary credential expiration time.
generation#
Generation of the ADLS storage connector
iam_role#
IAM role.
id#
Id of the storage connector uniquely identifying it in the Feature store.
password#
Password of the Snowflake storage connector
path#
If the connector refers to a path (e.g. S3) - return the path of the connector
role#
Role of the Snowflake storage connector
schema#
Schema of the Snowflake storage connector
secret_key#
Secret key.
server_encryption_algorithm#
Encryption algorithm if server-side S3 bucket encryption is enabled.
server_encryption_key#
Encryption key if server-side S3 bucket encryption is enabled.
service_credential#
Service credential of the ADLS storage connector
session_token#
Session token.
sf_options#
Additional options for the Snowflake storage connector
table#
Table of the Snowflake storage connector
table_name#
Table name for redshift cluster.
token#
OAuth token of the Snowflake storage connector
url#
URL of the Snowflake storage connector
user#
User of the Snowflake storage connector
warehouse#
Warehouse of the Snowflake storage connector
Methods#
snowflake_connector_options#
StorageConnector.snowflake_connector_options()
spark_options#
StorageConnector.spark_options()
Return prepared options to be passed to Spark, based on the additional arguments.
to_dict#
StorageConnector.to_dict()