Package com.logicalclocks.hsfs.spark

Class FeatureStore

java.lang.Object
com.logicalclocks.hsfs.FeatureStoreBase<Query>
com.logicalclocks.hsfs.spark.FeatureStore
public class FeatureStore extends FeatureStoreBase<Query>

  • Constructor Details

    • FeatureStore

      public FeatureStore()

  • Method Details

    • getFeatureGroup

      public FeatureGroup getFeatureGroup(@NonNull @NonNull String name, @NonNull @NonNull Integer version) throws FeatureStoreException, IOException
      Get a feature group object from the feature store.

      Feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getFeatureGroup("electricity_prices", 1);
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      Returns:
      FeatureGroup The feature group metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve feature group from the feature store.
      IOException - Generic IO exception.

    • getFeatureGroup

      public FeatureGroup getFeatureGroup(String name) throws FeatureStoreException, IOException
      Get a feature group object with default version `1` from the feature store.

      Feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getFeatureGroup("electricity_prices");
      Parameters:
      name - the name of the feature group
      Returns:
      FeatureGroup The feature group metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve feature group from the feature store.
      IOException - Generic IO exception.

    • getFeatureGroups

      public scala.collection.Seq<FeatureGroup> getFeatureGroups(@NonNull @NonNull String name) throws FeatureStoreException, IOException
      Get a list of all versions of a feature group from the feature store.

      Feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getFeatureGroups("electricity_prices");
      Parameters:
      name - the name of the feature group
      Returns:
      List of FeatureGroup metadata objects.
      Throws:
      FeatureStoreException - If unable to retrieve feature group from the feature store.
      IOException - Generic IO exception.

    • createFeatureGroup

      public FeatureGroup.FeatureGroupBuilder createFeatureGroup()
      Create feature group builder object. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup.FeatureGroupBuilder = fs.createFeatureGroup()
      Returns:
      FeatureGroup.FeatureGroupBuilder a FeatureGroup builder object.

    • getOrCreateFeatureGroup

      public FeatureGroup getOrCreateFeatureGroup(String name, Integer version) throws IOException, FeatureStoreException
      Get feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateFeatureGroup("fg_name", 1);
      Parameters:
      name - of the feature group to retrieve or create.
      version - of the feature group to retrieve or create.
      Returns:
      FeatureGroup The feature group metadata object.
      Throws:
      IOException - Generic IO exception.
      FeatureStoreException - If unable to retrieve FeatureGroup from the feature store.

    • getOrCreateFeatureGroup

      public FeatureGroup getOrCreateFeatureGroup(String name, Integer version, List<String> primaryKeys, boolean onlineEnabled, String eventTime) throws IOException, FeatureStoreException
      Get feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateFeatureGroup("fg_name", 1, primaryKeys, true, "datetime");
      Parameters:
      name - of the feature group to retrieve or create.
      version - of the feature group to retrieve or create.
      primaryKeys - A list of feature names to be used as primary key for the feature group. This primary key can be a composite key of multiple features and will be used as joining key.
      onlineEnabled - Define whether the feature group should be made available also in the online feature store for low latency access.
      eventTime - Name of the feature containing the event time for the features in this feature group. If eventTime is set the feature group can be used for point-in-time joins. The supported data types for the eventTime column are: timestamp, date and bigint
      Returns:
      FeatureGroup: The feature group metadata object
      Throws:
      IOException - Generic IO exception.
      FeatureStoreException - If unable to retrieve FeatureGroup from the feature store.

    • getOrCreateFeatureGroup

      public FeatureGroup getOrCreateFeatureGroup(String name, Integer version, List<String> primaryKeys, List<String> partitionKeys, boolean onlineEnabled, String eventTime) throws IOException, FeatureStoreException
      Get feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateFeatureGroup("fg_name", 1, primaryKeys, partitionKeys, true, "datetime");
      Parameters:
      name - of the feature group to retrieve or create.
      version - of the feature group to retrieve or create.
      primaryKeys - A list of feature names to be used as primary key for the feature group. This primary key can be a composite key of multiple features and will be used as joining key.
      partitionKeys - A list of feature names to be used as partition key when writing the feature data to the offline storage.
      onlineEnabled - Define whether the feature group should be made available also in the online feature store for low latency access.
      eventTime - Name of the feature containing the event time for the features in this feature group. If eventTime is set the feature group can be used for point-in-time joins. The supported data types for the eventTime column are: timestamp, date and bigint
      Returns:
      FeatureGroup: The feature group metadata object
      Throws:
      IOException - Generic IO exception.
      FeatureStoreException - If unable to retrieve FeatureGroup from the feature store.

    • getOrCreateFeatureGroup

      public FeatureGroup getOrCreateFeatureGroup(String name, Integer version, String description, List<String> primaryKeys, List<String> partitionKeys, String hudiPrecombineKey, boolean onlineEnabled, TimeTravelFormat timeTravelFormat, StatisticsConfig statisticsConfig, String topicName, String notificationTopicName, String eventTime, OnlineConfig onlineConfig) throws IOException, FeatureStoreException
      Get feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateFeatureGroup("fg_name", 1, primaryKeys, partitionKeys, true, "datetime");
      Parameters:
      name - of the feature group to retrieve or create.
      version - of the feature group to retrieve or create.
      description - contents of the feature group to improve discoverability for Data Scientists
      primaryKeys - A list of feature names to be used as primary key for the feature group. This primary key can be a composite key of multiple features and will be used as joining key.
      partitionKeys - A list of feature names to be used as partition key when writing the feature data to the offline storage.
      hudiPrecombineKey - A feature name to be used as a precombine key for the `HUDI` feature group. If feature group has time travel format `HUDI` and hudi precombine key was not specified then the first primary key of the feature group will be used as hudi precombine key.
      onlineEnabled - Define whether the feature group should be made available also in the online feature store for low latency access.
      timeTravelFormat - Format used for time travel: `TimeTravelFormat.HUDI` or `TimeTravelFormat.NONE`.
      statisticsConfig - A configuration object, to generally enable descriptive statistics computation for this feature group, `"correlations`" to turn on feature correlation computation, `"histograms"` to compute feature value frequencies and `"exact_uniqueness"` to compute uniqueness, distinctness and entropy. The values should be booleans indicating the setting. To fully turn off statistics computation pass `statisticsConfig=false`.
      topicName - Optionally, define the name of the topic used for data ingestion. If left undefined it defaults to using project topic.
      notificationTopicName - Optionally, define the name of the topic used for sending notifications when entries are inserted or updated on the online feature store. If left undefined no notifications are sent.
      eventTime - Name of the feature containing the event time for the features in this feature group. If eventTime is set the feature group can be used for point-in-time joins. The supported data types for the eventTime column are: timestamp, date and bigint
      onlineConfig - Optionally, define configuration which is used to configure online table.
      Returns:
      FeatureGroup: The feature group metadata object.
      Throws:
      IOException - Generic IO exception.
      FeatureStoreException - If unable to retrieve FeatureGroup from the feature store.

    • getStreamFeatureGroup

      public StreamFeatureGroup getStreamFeatureGroup(String name) throws FeatureStoreException, IOException
      Get a stream feature group object with default version `1` from the feature store.

      Getting a stream feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getStreamFeatureGroup("electricity_prices");
 <
      Specified by:
      getStreamFeatureGroup in class FeatureStoreBase<Query>
      Parameters:
      name - the name of the feature group
      Returns:
      FeatureGroup
      Throws:
      FeatureStoreException - If unable to retrieve feature group from the feature store.
      IOException - Generic IO exception.

    • getStreamFeatureGroup

      public StreamFeatureGroup getStreamFeatureGroup(@NonNull @NonNull String name, @NonNull @NonNull Integer version) throws FeatureStoreException, IOException
      Get a stream feature group object from the feature store.

      Getting a stream feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getStreamFeatureGroup("electricity_prices", 1);
      Specified by:
      getStreamFeatureGroup in class FeatureStoreBase<Query>
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      Returns:
      StreamFeatureGroup The stream feature group metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve feature group from the feature store.
      IOException - Generic IO exception.

    • createStreamFeatureGroup

      public StreamFeatureGroup createStreamFeatureGroup(@NonNull @NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKey, List<String> partitionKey, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, StorageConnector storageConnector, String path)
      Description copied from class: FeatureStoreBase
      Create a feature group object.
      Specified by:
      createStreamFeatureGroup in class FeatureStoreBase<Query>
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      description - descrption of the feature group
      onlineEnabled - whether the feature group should be online enabled
      timeTravelFormat - the data format to use to store the offline data
      primaryKey - list of primary keys
      partitionKey - list of partition keys
      eventTime - the feature/column to use as event time
      hudiPrecombineKey - if the timeTravelFormat is set to hudi, the feature/column to use as precombine key
      features - the list of feature objects if defined explicitly
      statisticsConfig - the statistics configuration for the feature group
      storageConnector - the storage connector to use to store the offline feature data (Default stored internally in Hopsworks).
      path - the path on the storage where to store the feature data.
      Returns:
      The feature group metadata object.

    • createStreamFeatureGroup

      public StreamFeatureGroup createStreamFeatureGroup(@NonNull @NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKey, List<String> partitionKey, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, DataSource dataSource)
      Description copied from class: FeatureStoreBase
      Create a stream feature group object.
      Specified by:
      createStreamFeatureGroup in class FeatureStoreBase<Query>
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      description - descrption of the feature group
      onlineEnabled - whether the feature group should be online enabled
      timeTravelFormat - the data format to use to store the offline data
      primaryKey - list of primary keys
      partitionKey - list of partition keys
      eventTime - the feature/column to use as event time
      hudiPrecombineKey - if the timeTravelFormat is set to hudi, the feature/column to use as precombine key
      features - the list of feature objects if defined explicitly
      statisticsConfig - the statistics configuration for the feature group
      dataSource - the data source specifying the location of the data.
      Returns:
      The feature group metadata object.

    • getOrCreateStreamFeatureGroup

      public StreamFeatureGroup getOrCreateStreamFeatureGroup(String name, Integer version) throws IOException, FeatureStoreException
      Get stream feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateStreamFeatureGroup("fg_name", 1);
      Parameters:
      name - of the feature group to retrieve or create.
      version - of the feature group to retrieve or create.
      Returns:
      FeatureGroup The feature group metadata object.
      Throws:
      IOException - Generic IO exception.
      FeatureStoreException - If unable to retrieve StreamFeatureGroup from the feature store.

    • getOrCreateStreamFeatureGroup

      public StreamFeatureGroup getOrCreateStreamFeatureGroup(String name, Integer version, List<String> primaryKeys, boolean onlineEnabled, String eventTime) throws IOException, FeatureStoreException
      Get stream feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateStreamFeatureGroup("fg_name", 1, primaryKeys, true, "datetime");
      Parameters:
      name - Name of the feature group to retrieve or create.
      version - Version of the feature group to retrieve or create.
      primaryKeys - A list of feature names to be used as primary key for the feature group. This primary key can be a composite key of multiple features and will be used as joining key.
      onlineEnabled - Define whether the feature group should be made available also in the online feature store for low latency access.
      eventTime - Name of the feature containing the event time for the features in this feature group. If eventTime is set the feature group can be used for point-in-time joins. The supported data types for the eventTime column are: timestamp, date and bigint
      Returns:
      FeatureGroup The feature group metadata object.
      Throws:
      IOException - Generic IO exception.
      FeatureStoreException - If unable to retrieve StreamFeatureGroup from the feature store.

    • getOrCreateStreamFeatureGroup

      public StreamFeatureGroup getOrCreateStreamFeatureGroup(String name, Integer version, List<String> primaryKeys, List<String> partitionKeys, boolean onlineEnabled, String eventTime) throws IOException, FeatureStoreException
      Get stream feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateStreamFeatureGroup("fg_name", 1, primaryKeys, partitionKeys, true,
        "datetime");
      Parameters:
      name - Name of the feature group to retrieve or create.
      version - Version of the feature group to retrieve or create.
      primaryKeys - A list of feature names to be used as primary key for the feature group. This primary key can be a composite key of multiple features and will be used as joining key.
      partitionKeys - A list of feature names to be used as partition key when writing the feature data to the offline storage.
      onlineEnabled - Define whether the feature group should be made available also in the online feature store for low latency access.
      eventTime - Name of the feature containing the event time for the features in this feature group. If eventTime is set the feature group can be used for point-in-time joins. The supported data types for the eventTime column are: timestamp, date and bigint
      Returns:
      FeatureGroup: The feature group metadata object
      Throws:
      IOException - Generic IO exception.
      FeatureStoreException - If unable to retrieve StreamFeatureGroup from the feature store.

    • getOrCreateStreamFeatureGroup

      public StreamFeatureGroup getOrCreateStreamFeatureGroup(String name, Integer version, String description, List<String> primaryKeys, List<String> partitionKeys, String hudiPrecombineKey, boolean onlineEnabled, StatisticsConfig statisticsConfig, String eventTime, TimeTravelFormat timeTravelFormat) throws IOException, FeatureStoreException
      Get stream feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StreamFeatureGroup fg = fs.getOrCreateStreamFeatureGroup("fg_name", 1, primaryKeys,
        partitionKeys, true, "datetime");
      Parameters:
      name - of the feature group to retrieve or create.
      version - of the feature group to retrieve or create.
      description - contents of the feature group to improve discoverability for Data Scientists
      primaryKeys - A list of feature names to be used as primary key for the feature group. This primary key can be a composite key of multiple features and will be used as joining key.
      partitionKeys - A list of feature names to be used as partition key when writing the feature data to the offline storage.
      hudiPrecombineKey - A feature name to be used as a precombine key for the `HUDI` feature group. If feature group has time travel format `HUDI` and hudi precombine key was not specified then the first primary key of the feature group will be used as hudi precombine key.
      onlineEnabled - Define whether the feature group should be made available also in the online feature store for low latency access.
      timeTravelFormat - Format used for time travel, defaults to `"HUDI"`.
      statisticsConfig - A configuration object, to generally enable descriptive statistics computation for this feature group, `"correlations`" to turn on feature correlation computation, `"histograms"` to compute feature value frequencies and `"exact_uniqueness"` to compute uniqueness, distinctness and entropy. The values should be booleans indicating the setting. To fully turn off statistics computation pass `statisticsConfig=false`.
      eventTime - Name of the feature containing the event time for the features in this feature group. If eventTime is set the feature group can be used for point-in-time joins.
      Returns:
      FeatureGroup: The feature group metadata object.
      Throws:
      IOException - Generic IO exception.
      FeatureStoreException - If unable to retrieve FeatureGroup from the feature store.

    • getOrCreateStreamFeatureGroup

      public StreamFeatureGroup getOrCreateStreamFeatureGroup(@NonNull @NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKeys, List<String> partitionKeys, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, StorageConnector storageConnector, String path, OnlineConfig onlineConfig) throws IOException, FeatureStoreException
      Get stream feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StreamFeatureGroup fg = fs.getOrCreateStreamFeatureGroup("fg_name", 1, primaryKeys,
        partitionKeys, hudiPrecombineKey, true,  statisticsConfig, "datetime", timeTravelFormat,
        features, storageConnector, path, onlineConfig);
      Specified by:
      getOrCreateStreamFeatureGroup in class FeatureStoreBase<Query>
      Parameters:
      name - of the feature group to retrieve or create.
      version - of the feature group to retrieve or create.
      description - contents of the feature group to improve discoverability for Data Scientists
      primaryKeys - A list of feature names to be used as primary key for the feature group. This primary key can be a composite key of multiple features and will be used as joining key.
      partitionKeys - A list of feature names to be used as partition key when writing the feature data to the offline storage.
      hudiPrecombineKey - A feature name to be used as a precombine key for the `HUDI` feature group. If feature group has time travel format `HUDI` and hudi precombine key was not specified then the first primary key of the feature group will be used as hudi precombine key.
      onlineEnabled - Define whether the feature group should be made available also in the online feature store for low latency access.
      timeTravelFormat - Format used for time travel, defaults to `"HUDI"`.
      statisticsConfig - A configuration object, to generally enable descriptive statistics computation for this feature group, `"correlations`" to turn on feature correlation computation, `"histograms"` to compute feature value frequencies and `"exact_uniqueness"` to compute uniqueness, distinctness and entropy. The values should be booleans indicating the setting. To fully turn off statistics computation pass `statisticsConfig=false`.
      eventTime - Name of the feature containing the event time for the features in this feature group. If eventTime is set the feature group can be used for point-in-time joins.
      onlineConfig - Optionally, define configuration which is used to configure online table.
      features - the list of feature objects if defined explicitly
      storageConnector - the storage connector to use to store the offline feature data (Default stored internally in Hopsworks).
      path - the path on the storage where to store the feature data.
      Returns:
      FeatureGroup: The feature group metadata object.
      Throws:
      IOException - Generic IO exception.
      FeatureStoreException - If unable to retrieve FeatureGroup from the feature store.

    • getOrCreateStreamFeatureGroup

      public StreamFeatureGroup getOrCreateStreamFeatureGroup(@NonNull @NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKeys, List<String> partitionKeys, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, DataSource dataSource, OnlineConfig onlineConfig) throws IOException, FeatureStoreException
      Description copied from class: FeatureStoreBase
      Get a stream feature group metadata object or create a new one if it doesn't exist.
      Specified by:
      getOrCreateStreamFeatureGroup in class FeatureStoreBase<Query>
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      description - descrption of the feature group
      onlineEnabled - whether the feature group should be online enabled
      timeTravelFormat - the data format to use to store the offline data
      primaryKeys - list of primary keys
      partitionKeys - list of partition keys
      eventTime - the feature/column to use as event time
      hudiPrecombineKey - if the timeTravelFormat is set to hudi, the feature/column to use as precombine key
      features - the list of feature objects if defined explicitly
      statisticsConfig - the statistics configuration for the feature group
      dataSource - the data source specifying the location of the data.
      onlineConfig - the online configuration for the feature group.
      Returns:
      The feature group metadata object.
      Throws:
      IOException
      FeatureStoreException

    • createExternalFeatureGroup

      public ExternalFeatureGroup.ExternalFeatureGroupBuilder createExternalFeatureGroup()
      Create external feature group builder object. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        ExternalFeatureGroup.ExternalFeatureGroupBuilder = fs.createExternalFeatureGroup()
      Returns:
      ExternalFeatureGroup.ExternalFeatureGroupBuilder a ExternalFeatureGroup builder object.

    • createOnDemandFeatureGroup

      @Deprecated public ExternalFeatureGroup.ExternalFeatureGroupBuilder createOnDemandFeatureGroup()
      Deprecated.

    • getExternalFeatureGroups

      public scala.collection.Seq<ExternalFeatureGroup> getExternalFeatureGroups(@NonNull @NonNull String name) throws FeatureStoreException, IOException
      Get a list of all versions of an external feature group from the feature store.

      Feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getExternalFeatureGroups("external_fg_name");
      Parameters:
      name - The name of the feature group.
      Returns:
      List of ExternalFeatureGroup metadata objects.
      Throws:
      FeatureStoreException - If unable to retrieve feature group from the feature store.
      IOException - Generic IO exception.

    • getExternalFeatureGroup

      public ExternalFeatureGroup getExternalFeatureGroup(@NonNull @NonNull String name, @NonNull @NonNull Integer version) throws FeatureStoreException, IOException
      Get an external feature group object from the feature store.

      Getting a stream feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        ExternalFeatureGroup fg = fs.getExternalFeatureGroup("external_fg_name", 1);
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      Returns:
      ExternalFeatureGroup The external feature group metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve feature group from the feature store.
      IOException - Generic IO exception.

    • getExternalFeatureGroup

      public ExternalFeatureGroup getExternalFeatureGroup(String name) throws FeatureStoreException, IOException
      Get an external feature group object with default version `1` from the feature store.

      Getting external feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        ExternalFeatureGroup fg = fs.getExternalFeatureGroup("external_fg_name");
      Parameters:
      name - the name of the feature group
      Returns:
      ExternalFeatureGroup The external feature group metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve feature group from the feature store.
      IOException - Generic IO exception.

    • getStorageConnector

      public StorageConnector getStorageConnector(String name) throws FeatureStoreException, IOException
      Get a previously created storage connector from the feature store.

      Storage connectors encapsulate all information needed for the execution engine to read and write to a specific storage.

      If you want to connect to the online feature store, see the getOnlineStorageConnector` method to get the JDBC connector for the Online Feature Store. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector sc = fs.getStorageConnector("sc_name");
      Overrides:
      getStorageConnector in class FeatureStoreBase<Query>
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getHopsFsConnector

      public StorageConnector.HopsFsConnector getHopsFsConnector(String name) throws FeatureStoreException, IOException
      Get a previously created HopsFs compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.HopsFsConnector hfsSc = fs.getHopsFsConnector("hfs_sc_name");
      Overrides:
      getHopsFsConnector in class FeatureStoreBase<Query>
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.HopsFsConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getJdbcConnector

      public StorageConnector.JdbcConnector getJdbcConnector(String name) throws FeatureStoreException, IOException
      Get a previously created JDBC compliant storage connector from the feature store.

      If you want to connect to the online feature store, see the getOnlineStorageConnector` method to get the JDBC connector for the Online Feature Store. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.JdbcConnector jdbcSc = fs.getJdbcConnector("jdbc_sc_name");
      Overrides:
      getJdbcConnector in class FeatureStoreBase<Query>
      Parameters:
      name - Name of the jdbc storage connector to retrieve.
      Returns:
      StorageConnector.JdbcConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getOnlineStorageConnector

      public StorageConnector.JdbcConnector getOnlineStorageConnector() throws FeatureStoreException, IOException
      Get a previously created JDBC compliant storage connector from the feature store to connect to the online feature store. 
       
        //get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.JdbcConnector onlineSc = fs.getOnlineStorageConnector("online_sc_name");
      Overrides:
      getOnlineStorageConnector in class FeatureStoreBase<Query>
      Returns:
      StorageConnector.JdbcConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getS3Connector

      public StorageConnector.S3Connector getS3Connector(String name) throws FeatureStoreException, IOException
      Get a previously created S3 compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.S3Connector s3Sc = fs.getS3Connector("s3_sc_name");
      Overrides:
      getS3Connector in class FeatureStoreBase<Query>
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.S3Connector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getRedshiftConnector

      public StorageConnector.RedshiftConnector getRedshiftConnector(String name) throws FeatureStoreException, IOException
      Get a previously created Redshift compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.RedshiftConnector rshSc = fs.getRedshiftConnector("rsh_sc_name");
      Overrides:
      getRedshiftConnector in class FeatureStoreBase<Query>
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.RedshiftConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getSnowflakeConnector

      public StorageConnector.SnowflakeConnector getSnowflakeConnector(String name) throws FeatureStoreException, IOException
      Get a previously created Snowflake compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.SnowflakeConnector snflSc = fs.getSnowflakeConnector("snfl_sc_name");
      Overrides:
      getSnowflakeConnector in class FeatureStoreBase<Query>
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.SnowflakeConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getAdlsConnector

      public StorageConnector.AdlsConnector getAdlsConnector(String name) throws FeatureStoreException, IOException
      Get a previously created Adls compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.AdlsConnectorr adlslSc = fs.getAdlsConnector("adls_sc_name");
      Overrides:
      getAdlsConnector in class FeatureStoreBase<Query>
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.AdlsConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getKafkaConnector

      public StorageConnector.KafkaConnector getKafkaConnector(String name) throws FeatureStoreException, IOException
      Get a previously created Kafka compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.KafkaConnector kafkaSc = fs.getKafkaConnector("kafka_sc_name");
      Overrides:
      getKafkaConnector in class FeatureStoreBase<Query>
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.KafkaConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getBigqueryConnector

      public StorageConnector.BigqueryConnector getBigqueryConnector(String name) throws FeatureStoreException, IOException
      Get a previously created BigQuery compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.BigqueryConnector bigqSc = fs.getBigqueryConnector("bigq_sc_name");
      Overrides:
      getBigqueryConnector in class FeatureStoreBase<Query>
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.BigqueryConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getGcsConnector

      public StorageConnector.GcsConnector getGcsConnector(String name) throws FeatureStoreException, IOException
      Get a previously created Gcs compliant storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.GcsConnector gcsSc = fs.getGcsConnector("gsc_sc_name");
      Overrides:
      getGcsConnector in class FeatureStoreBase<Query>
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.GcsConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getSqlConnector

      public StorageConnector.SqlConnector getSqlConnector(String name) throws FeatureStoreException, IOException
      Get a previously created SQL storage connector from the feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.SqlConnector sqlSc = fs.getSqlConnector("sql_sc_name");
      Specified by:
      getSqlConnector in class FeatureStoreBase<Query>
      Parameters:
      name - Name of the storage connector to retrieve.
      Returns:
      StorageConnector.SqlConnector Storage connector object.
      Throws:
      FeatureStoreException - If unable to retrieve StorageConnector from the feature store.
      IOException - Generic IO exception.

    • getOnDemandFeatureGroup

      @Deprecated public ExternalFeatureGroup getOnDemandFeatureGroup(@NonNull @NonNull String name, @NonNull @NonNull Integer version) throws FeatureStoreException, IOException
      Deprecated.
      Throws:
      FeatureStoreException
      IOException

    • getOnDemandFeatureGroup

      @Deprecated public ExternalFeatureGroup getOnDemandFeatureGroup(String name) throws FeatureStoreException, IOException
      Deprecated.
      Throws:
      FeatureStoreException
      IOException

    • getOnDemandFeatureGroups

      @Deprecated public scala.collection.Seq<ExternalFeatureGroup> getOnDemandFeatureGroups(@NonNull @NonNull String name) throws FeatureStoreException, IOException
      Deprecated.
      Throws:
      FeatureStoreException
      IOException

    • createFeatureView

      public FeatureView.FeatureViewBuilder createFeatureView()

    • getOrCreateFeatureView

      public FeatureView getOrCreateFeatureView(String name, Query query, Integer version) throws FeatureStoreException, IOException
      Get feature view metadata object or create a new one if it doesn't exist. This method doesn't update existing feature view metadata. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureView fv = fs.getOrCreateFeatureView("fv_name", query, 1);
      Parameters:
      name - Name of the feature view.
      query - Query object.
      version - Version of the feature view.
      Returns:
      FeatureView The feature view metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve FeatureView from the feature store.
      IOException - Generic IO exception.

    • getOrCreateFeatureView

      public FeatureView getOrCreateFeatureView(String name, Query query, Integer version, String description, List<String> labels) throws FeatureStoreException, IOException
      Get feature view metadata object or create a new one if it doesn't exist. This method doesn't update existing feature view metadata. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureView fv = fs.getOrCreateFeatureView("fv_name", query, 1, description, labels);
        "datetime");
      Parameters:
      name - Name of the feature view.
      query - Query object.
      version - Version of the feature view.
      description - Description of the feature view.
      labels - List of label features.
      Returns:
      FeatureView The feature view metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve FeatureView from the feature store.
      IOException - Generic IO exception.

    • getFeatureView

      public FeatureView getFeatureView(@NonNull @NonNull String name, @NonNull @NonNull Integer version) throws FeatureStoreException, IOException
      Get a feature view object from the selected feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureView fv = fs.getFeatureView("fv_name", 1);
      Specified by:
      getFeatureView in class FeatureStoreBase<Query>
      Parameters:
      name - Name of the feature view.
      version - Version to get.
      Returns:
      FeatureView The feature view metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve FeatureView from the feature store.
      IOException - Generic IO exception.

    • getFeatureView

      public FeatureView getFeatureView(String name) throws FeatureStoreException, IOException
      Get a feature view object with the default version `1` from the selected feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureView fv = fs.getFeatureView("fv_name");
      Specified by:
      getFeatureView in class FeatureStoreBase<Query>
      Parameters:
      name - Name of the feature view.
      Returns:
      FeatureView The feature view metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve FeatureView from the feature store.
      IOException - Generic IO exception.

    • sql

      public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> sql(String query)

    • createTrainingDataset

      @Deprecated public TrainingDataset.TrainingDatasetBuilder createTrainingDataset()
      Deprecated.

    • getTrainingDataset

      @Deprecated public TrainingDataset getTrainingDataset(@NonNull @NonNull String name, @NonNull @NonNull Integer version) throws FeatureStoreException, IOException
      Deprecated.
      Throws:
      FeatureStoreException
      IOException

    • getTrainingDataset

      @Deprecated public TrainingDataset getTrainingDataset(String name) throws FeatureStoreException, IOException
      Deprecated.
      Throws:
      FeatureStoreException
      IOException

    • getTrainingDatasets

      @Deprecated public scala.collection.Seq<TrainingDataset> getTrainingDatasets(@NonNull @NonNull String name) throws FeatureStoreException, IOException
      Deprecated.
      Throws:
      FeatureStoreException
      IOException