Package com.logicalclocks.hsfs

Class FeatureStoreBase<T2 extends QueryBase>

java.lang.Object
com.logicalclocks.hsfs.FeatureStoreBase<T2>
Direct Known Subclasses:
FeatureStore, FeatureStore, FeatureStore, FeatureStore
public abstract class FeatureStoreBase<T2 extends QueryBase> extends Object

  • Field Details

    • featureGroupApi

      protected FeatureGroupApi featureGroupApi

    • trainingDatasetApi

      protected TrainingDatasetApi trainingDatasetApi

    • storageConnectorApi

      protected StorageConnectorApi storageConnectorApi

    • LOGGER

      protected static final org.slf4j.Logger LOGGER

    • DEFAULT_VERSION

      protected static final Integer DEFAULT_VERSION

  • Constructor Details

    • FeatureStoreBase

      public FeatureStoreBase()

  • Method Details

    • createStreamFeatureGroup

      @Deprecated public abstract FeatureGroupBase createStreamFeatureGroup(@NonNull @NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKeys, List<String> partitionKeys, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, StorageConnector storageConnector, String path)
      Deprecated.
      Use createStreamFeatureGroup(String, Integer, String, Boolean, TimeTravelFormat, List, List, String, String, List, StatisticsConfig, DataSource) instead.
      Create a feature group object.
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      description - descrption of the feature group
      onlineEnabled - whether the feature group should be online enabled
      timeTravelFormat - the data format to use to store the offline data
      primaryKeys - list of primary keys
      partitionKeys - list of partition keys
      eventTime - the feature/column to use as event time
      hudiPrecombineKey - if the timeTravelFormat is set to hudi, the feature/column to use as precombine key
      features - the list of feature objects if defined explicitly
      statisticsConfig - the statistics configuration for the feature group
      storageConnector - the storage connector to use to store the offline feature data (Default stored internally in Hopsworks).
      path - the path on the storage where to store the feature data.
      Returns:
      The feature group metadata object.

    • createStreamFeatureGroup

      public abstract FeatureGroupBase createStreamFeatureGroup(@NonNull @NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKeys, List<String> partitionKeys, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, DataSource dataSource)
      Create a stream feature group object.
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      description - descrption of the feature group
      onlineEnabled - whether the feature group should be online enabled
      timeTravelFormat - the data format to use to store the offline data
      primaryKeys - list of primary keys
      partitionKeys - list of partition keys
      eventTime - the feature/column to use as event time
      hudiPrecombineKey - if the timeTravelFormat is set to hudi, the feature/column to use as precombine key
      features - the list of feature objects if defined explicitly
      statisticsConfig - the statistics configuration for the feature group
      dataSource - the data source specifying the location of the data.
      Returns:
      The feature group metadata object.

    • getOrCreateStreamFeatureGroup

      @Deprecated public abstract FeatureGroupBase getOrCreateStreamFeatureGroup(@NonNull @NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKeys, List<String> partitionKeys, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, StorageConnector storageConnector, String path, OnlineConfig onlineConfig) throws IOException, FeatureStoreException
      Deprecated.
      Use getOrCreateStreamFeatureGroup(String, Integer, String, Boolean, TimeTravelFormat, List, List, String, String, List, StatisticsConfig, DataSource, OnlineConfig) instead.
      Get a feature group metadata object or create a new one if it doesn't exists.
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      description - descrption of the feature group
      onlineEnabled - whether the feature group should be online enabled
      timeTravelFormat - the data format to use to store the offline data
      primaryKeys - list of primary keys
      partitionKeys - list of partition keys
      eventTime - the feature/column to use as event time
      hudiPrecombineKey - if the timeTravelFormat is set to hudi, the feature/column to use as precombine key
      features - the list of feature objects if defined explicitly
      statisticsConfig - the statistics configuration for the feature group
      storageConnector - the storage connector to use to store the offline feature data (Default stored internally in Hopsworks).
      path - the path on the storage where to store the feature data.
      Returns:
      The feature group metadata object.
      Throws:
      IOException
      FeatureStoreException

    • getOrCreateStreamFeatureGroup

      public abstract FeatureGroupBase getOrCreateStreamFeatureGroup(@NonNull @NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKeys, List<String> partitionKeys, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, DataSource dataSource, OnlineConfig onlineConfig) throws IOException, FeatureStoreException
      Get a stream feature group metadata object or create a new one if it doesn't exist.
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      description - descrption of the feature group
      onlineEnabled - whether the feature group should be online enabled
      timeTravelFormat - the data format to use to store the offline data
      primaryKeys - list of primary keys
      partitionKeys - list of partition keys
      eventTime - the feature/column to use as event time
      hudiPrecombineKey - if the timeTravelFormat is set to hudi, the feature/column to use as precombine key
      features - the list of feature objects if defined explicitly
      statisticsConfig - the statistics configuration for the feature group
      dataSource - the data source specifying the location of the data.
      onlineConfig - the online configuration for the feature group.
      Returns:
      The feature group metadata object.
      Throws:
      IOException
      FeatureStoreException

    • getStreamFeatureGroup

      public abstract Object getStreamFeatureGroup(@NonNull @NonNull String name, @NonNull @NonNull Integer version) throws FeatureStoreException, IOException
      Get a feature group object from the feature store.
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      Returns:
      FeatureGroup
      Throws:
      FeatureStoreException
      IOException

    • getStreamFeatureGroup

      public abstract Object getStreamFeatureGroup(String name) throws FeatureStoreException, IOException
      Get a feature group object with default version `1` from the feature store.
      Parameters:
      name - the name of the feature group
      Returns:
      FeatureGroup
      Throws:
      FeatureStoreException
      IOException

    • getFeatureView

      public abstract Object getFeatureView(String name) throws FeatureStoreException, IOException
      Throws:
      FeatureStoreException
      IOException

    • getFeatureView

      public abstract Object getFeatureView(@NonNull @NonNull String name, @NonNull @NonNull Integer version) throws FeatureStoreException, IOException
      Throws:
      FeatureStoreException
      IOException

    • getStorageConnector

      public StorageConnector getStorageConnector(String name) throws FeatureStoreException, IOException
      Get a previously created storage connector from the feature store.

      Storage connectors encapsulate all information needed for the execution engine to read and write to a specific storage.

      If you want to connect to the online feature store, see the getOnlineStorageConnector` method to get the JDBC connector for the Online Feature Store. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector sc = fs.getStorageConnector("sc_name");
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getDataSource

      public DataSource getDataSource(String name) throws FeatureStoreException, IOException
      Get a previously created data source from the feature store.

      data sources encapsulate all information needed for the execution engine to read and write to a specific storage.

      If you want to connect to the online feature store, see the getOnlineDataSource` method to get the JDBC connector for the Online Feature Store. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        DataSource ds = fs.getDataSource("ds_name");
      Parameters:
      name - Name of the data source to retrieve.
      Returns:
      DataSource Data source object.
      Throws:
      FeatureStoreException - If unable to retrieve DataSource from the feature store.
      IOException - Generic IO exception.

    • getHopsFsConnector

      public StorageConnector.HopsFsConnector getHopsFsConnector(String name) throws FeatureStoreException, IOException
      Get a previously created HopsFs compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.HopsFsConnector hfsSc = fs.getHopsFsConnector("hfs_sc_name");
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.HopsFsConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getJdbcConnector

      public StorageConnector.JdbcConnector getJdbcConnector(String name) throws FeatureStoreException, IOException
      Get a previously created JDBC compliant storage connector from the feature store.

      If you want to connect to the online feature store, see the getOnlineStorageConnector` method to get the JDBC connector for the Online Feature Store. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.JdbcConnector jdbcSc = fs.getJdbcConnector("jdbc_sc_name");
      Parameters:
      name - Name of the jdbc storage connector to retrieve.
      Returns:
      StorageConnector.JdbcConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getSqlConnector

      public abstract Object getSqlConnector(String name) throws FeatureStoreException, IOException
      Throws:
      FeatureStoreException
      IOException

    • getOnlineStorageConnector

      public StorageConnector.JdbcConnector getOnlineStorageConnector() throws FeatureStoreException, IOException
      Get a previously created JDBC compliant storage connector from the feature store to connect to the online feature store. 
       
        //get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.JdbcConnector onlineSc = fs.getOnlineStorageConnector("online_sc_name");
      Returns:
      StorageConnector.JdbcConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getS3Connector

      public StorageConnector.S3Connector getS3Connector(String name) throws FeatureStoreException, IOException
      Get a previously created S3 compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.S3Connector s3Sc = fs.getS3Connector("s3_sc_name");
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.S3Connector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getRedshiftConnector

      public StorageConnector.RedshiftConnector getRedshiftConnector(String name) throws FeatureStoreException, IOException
      Get a previously created Redshift compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.RedshiftConnector rshSc = fs.getRedshiftConnector("rsh_sc_name");
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.RedshiftConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getSnowflakeConnector

      public StorageConnector.SnowflakeConnector getSnowflakeConnector(String name) throws FeatureStoreException, IOException
      Get a previously created Snowflake compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.SnowflakeConnector snflSc = fs.getSnowflakeConnector("snfl_sc_name");
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.SnowflakeConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getAdlsConnector

      public StorageConnector.AdlsConnector getAdlsConnector(String name) throws FeatureStoreException, IOException
      Get a previously created Adls compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.AdlsConnectorr adlslSc = fs.getAdlsConnector("adls_sc_name");
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.AdlsConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getKafkaConnector

      public StorageConnector.KafkaConnector getKafkaConnector(String name) throws FeatureStoreException, IOException
      Get a previously created Kafka compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.KafkaConnector kafkaSc = fs.getKafkaConnector("kafka_sc_name");
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.KafkaConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getBigqueryConnector

      public StorageConnector.BigqueryConnector getBigqueryConnector(String name) throws FeatureStoreException, IOException
      Get a previously created BigQuery compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.BigqueryConnector bigqSc = fs.getBigqueryConnector("bigq_sc_name");
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.BigqueryConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getGcsConnector

      public StorageConnector.GcsConnector getGcsConnector(String name) throws FeatureStoreException, IOException
      Get a previously created Gcs compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.GcsConnector gcsSc = fs.getGcsConnector("gsc_sc_name");
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.GcsConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • toString

      public String toString()
      Overrides:
      toString in class Object