Package com.logicalclocks.hsfs.beam

Class FeatureStore

java.lang.Object
com.logicalclocks.hsfs.FeatureStoreBase<com.logicalclocks.hsfs.beam.constructor.Query>
com.logicalclocks.hsfs.beam.FeatureStore
public class FeatureStore extends FeatureStoreBase<com.logicalclocks.hsfs.beam.constructor.Query>

  • Constructor Details

    • FeatureStore

      public FeatureStore()

  • Method Details

    • createStreamFeatureGroup

      public StreamFeatureGroup.StreamFeatureGroupBuilder createStreamFeatureGroup()
      Create a feature group builder object. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();

        // create feature group metadata object
        StreamFeatureGroup streamFeatureGroup = fs.createStreamFeatureGroup()
               .name("documentation_example")
               .version(1)
               .primaryKeys(Collections.singletonList("pk"))
               .eventTime("event_time")
               .onlineEnabled(true)
               .features(features)
               .build();

         // save the feature group metadata object on the feature store
         streamFeatureGroup.save()
      Returns:
      StreamFeatureGroup.StreamFeatureGroupBuilder a StreamFeatureGroup builder object.

    • createStreamFeatureGroup

      public StreamFeatureGroup createStreamFeatureGroup(@NonNull @NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKeys, List<String> partitionKeys, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, StorageConnector storageConnector, String path)
      Description copied from class: FeatureStoreBase
      Create a feature group object.
      Specified by:
      createStreamFeatureGroup in class FeatureStoreBase<com.logicalclocks.hsfs.beam.constructor.Query>
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      description - descrption of the feature group
      onlineEnabled - whether the feature group should be online enabled
      timeTravelFormat - the data format to use to store the offline data
      primaryKeys - list of primary keys
      partitionKeys - list of partition keys
      eventTime - the feature/column to use as event time
      hudiPrecombineKey - if the timeTravelFormat is set to hudi, the feature/column to use as precombine key
      features - the list of feature objects if defined explicitly
      statisticsConfig - the statistics configuration for the feature group
      storageConnector - the storage connector to use to store the offline feature data (Default stored internally in Hopsworks).
      path - the path on the storage where to store the feature data.
      Returns:
      The feature group metadata object.

    • createStreamFeatureGroup

      public StreamFeatureGroup createStreamFeatureGroup(@NonNull @NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKeys, List<String> partitionKeys, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, DataSource dataSource)
      Description copied from class: FeatureStoreBase
      Create a stream feature group object.
      Specified by:
      createStreamFeatureGroup in class FeatureStoreBase<com.logicalclocks.hsfs.beam.constructor.Query>
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      description - descrption of the feature group
      onlineEnabled - whether the feature group should be online enabled
      timeTravelFormat - the data format to use to store the offline data
      primaryKeys - list of primary keys
      partitionKeys - list of partition keys
      eventTime - the feature/column to use as event time
      hudiPrecombineKey - if the timeTravelFormat is set to hudi, the feature/column to use as precombine key
      features - the list of feature objects if defined explicitly
      statisticsConfig - the statistics configuration for the feature group
      dataSource - the data source specifying the location of the data.
      Returns:
      The feature group metadata object.

    • getOrCreateStreamFeatureGroup

      public StreamFeatureGroup getOrCreateStreamFeatureGroup(@NonNull @NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKeys, List<String> partitionKeys, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, StorageConnector storageConnector, String path, OnlineConfig onlineConfig) throws IOException, FeatureStoreException
      Description copied from class: FeatureStoreBase
      Get a feature group metadata object or create a new one if it doesn't exists.
      Specified by:
      getOrCreateStreamFeatureGroup in class FeatureStoreBase<com.logicalclocks.hsfs.beam.constructor.Query>
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      description - descrption of the feature group
      onlineEnabled - whether the feature group should be online enabled
      timeTravelFormat - the data format to use to store the offline data
      primaryKeys - list of primary keys
      partitionKeys - list of partition keys
      eventTime - the feature/column to use as event time
      hudiPrecombineKey - if the timeTravelFormat is set to hudi, the feature/column to use as precombine key
      features - the list of feature objects if defined explicitly
      statisticsConfig - the statistics configuration for the feature group
      storageConnector - the storage connector to use to store the offline feature data (Default stored internally in Hopsworks).
      path - the path on the storage where to store the feature data.
      Returns:
      The feature group metadata object.
      Throws:
      IOException
      FeatureStoreException

    • getOrCreateStreamFeatureGroup

      public StreamFeatureGroup getOrCreateStreamFeatureGroup(@NonNull @NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKeys, List<String> partitionKeys, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, DataSource dataSource, OnlineConfig onlineConfig) throws IOException, FeatureStoreException
      Description copied from class: FeatureStoreBase
      Get a stream feature group metadata object or create a new one if it doesn't exist.
      Specified by:
      getOrCreateStreamFeatureGroup in class FeatureStoreBase<com.logicalclocks.hsfs.beam.constructor.Query>
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      description - descrption of the feature group
      onlineEnabled - whether the feature group should be online enabled
      timeTravelFormat - the data format to use to store the offline data
      primaryKeys - list of primary keys
      partitionKeys - list of partition keys
      eventTime - the feature/column to use as event time
      hudiPrecombineKey - if the timeTravelFormat is set to hudi, the feature/column to use as precombine key
      features - the list of feature objects if defined explicitly
      statisticsConfig - the statistics configuration for the feature group
      dataSource - the data source specifying the location of the data.
      onlineConfig - the online configuration for the feature group.
      Returns:
      The feature group metadata object.
      Throws:
      IOException
      FeatureStoreException

    • getStreamFeatureGroup

      public Object getStreamFeatureGroup(String name) throws FeatureStoreException, IOException
      Get a stream feature group object from the feature store.

      Getting a stream feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getStreamFeatureGroup("electricity_prices", 1);
      Specified by:
      getStreamFeatureGroup in class FeatureStoreBase<com.logicalclocks.hsfs.beam.constructor.Query>
      Parameters:
      name - the name of the feature group
      Returns:
      StreamFeatureGroup The stream feature group metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve feature group from the feature store.
      IOException - Generic IO exception.

    • getStreamFeatureGroup

      public StreamFeatureGroup getStreamFeatureGroup(@NonNull @NonNull String name, @NonNull @NonNull Integer version) throws FeatureStoreException, IOException
      Get a stream feature group object from the feature store.

      Getting a stream feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views. 

       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getStreamFeatureGroup("electricity_prices", 1);
      Specified by:
      getStreamFeatureGroup in class FeatureStoreBase<com.logicalclocks.hsfs.beam.constructor.Query>
      Parameters:
      name - the name of the feature group
      version - the version of the feature group
      Returns:
      StreamFeatureGroup The stream feature group metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve feature group from the feature store.
      IOException - Generic IO exception.

    • getFeatureView

      public FeatureView getFeatureView(@NonNull @NonNull String name, @NonNull @NonNull Integer version) throws FeatureStoreException, IOException
      Get a feature view object from the selected feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureView fv = fs.getFeatureView("fv_name", 1);
      Specified by:
      getFeatureView in class FeatureStoreBase<com.logicalclocks.hsfs.beam.constructor.Query>
      Parameters:
      name - Name of the feature view.
      version - Version to get.
      Returns:
      FeatureView The feature view metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve FeatureView from the feature store.
      IOException - Generic IO exception.

    • getFeatureView

      public FeatureView getFeatureView(String name) throws FeatureStoreException, IOException
      Get a feature view object from the selected feature store. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureView fv = fs.getFeatureView("fv_name", 1);
      Specified by:
      getFeatureView in class FeatureStoreBase<com.logicalclocks.hsfs.beam.constructor.Query>
      Parameters:
      name - Name of the feature view.
      Returns:
      FeatureView The feature view metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve FeatureView from the feature store.
      IOException - Generic IO exception.

    • createFeatureView

      public FeatureView.FeatureViewBuilder createFeatureView()
      Create a new feature view metadata object. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureView fv = fs.createFeatureView
          .name("fv_name")
          .version(1)
          .query(query)
          .build() // The build method also save the feature view metadata to Hopsworks
      Returns:
      FeatureView.FeatureViewBuilder Feature View Builder object to build the feature view metadata object

    • getOrCreateFeatureView

      public FeatureView getOrCreateFeatureView(String name, com.logicalclocks.hsfs.beam.constructor.Query query, Integer version) throws FeatureStoreException, IOException
      Get feature view metadata object or create a new one if it doesn't exist. This method doesn't update existing feature view metadata. 
       
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureView fv = fs.getOrCreateFeatureView("fv_name", query, 1);
      Parameters:
      name - Name of the feature view.
      query - Query object.
      version - Version of the feature view.
      Returns:
      FeatureView The feature view metadata object.
      Throws:
      FeatureStoreException - If unable to retrieve FeatureView from the feature store.
      IOException - Generic IO exception.

    • getSqlConnector

      public StorageConnector.SqlConnector getSqlConnector(String name) throws FeatureStoreException, IOException
      Specified by:
      getSqlConnector in class FeatureStoreBase<com.logicalclocks.hsfs.beam.constructor.Query>
      Throws:
      FeatureStoreException
      IOException