com.logicalclocks.hsfs

Class FeatureStoreBase<T2 extends QueryBase>

java.lang.Object

com.logicalclocks.hsfs.FeatureStoreBase<T2>

Direct Known Subclasses:

FeatureStore, FeatureStore, FeatureStore, FeatureStore

public abstract class FeatureStoreBase<T2 extends QueryBase>
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
protected static Integer	DEFAULT_VERSION
protected FeatureGroupApi	featureGroupApi
protected static org.slf4j.Logger	LOGGER
protected StorageConnectorApi	storageConnectorApi
protected TrainingDatasetApi	trainingDatasetApi

Constructor Summary

Constructors

Constructor and Description
FeatureStoreBase()

Method Summary

Modifier and Type	Method and Description
abstract FeatureGroupBase	createStreamFeatureGroup(@NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKeys, List<String> partitionKeys, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, StorageConnector storageConnector, String path) Create a feature group object.
StorageConnector.AdlsConnector	getAdlsConnector(String name) Get a previously created Adls compliant storage connector from the feature store.
StorageConnector.BigqueryConnector	getBigqueryConnector(String name) Get a previously created BigQuery compliant storage connector from the feature store.
abstract Object	getFeatureView(String name)
abstract Object	getFeatureView(@NonNull String name, @NonNull Integer version)
StorageConnector.GcsConnector	getGcsConnector(String name) Get a previously created Gcs compliant storage connector from the feature store.
StorageConnector.HopsFsConnector	getHopsFsConnector(String name) Get a previously created HopsFs compliant storage connector from the feature store.
StorageConnector.JdbcConnector	getJdbcConnector(String name) Get a previously created JDBC compliant storage connector from the feature store.
StorageConnector.KafkaConnector	getKafkaConnector(String name) Get a previously created Kafka compliant storage connector from the feature store.
StorageConnector.JdbcConnector	getOnlineStorageConnector() Get a previously created JDBC compliant storage connector from the feature store to connect to the online feature store.
abstract FeatureGroupBase	getOrCreateStreamFeatureGroup(@NonNull String name, Integer version, String description, Boolean onlineEnabled, TimeTravelFormat timeTravelFormat, List<String> primaryKeys, List<String> partitionKeys, String eventTime, String hudiPrecombineKey, List<Feature> features, StatisticsConfig statisticsConfig, StorageConnector storageConnector, String path) Get a feature group metadata object or create a new one if it doesn't exists.
StorageConnector.RedshiftConnector	getRedshiftConnector(String name) Get a previously created Redshift compliant storage connector from the feature store.
StorageConnector.S3Connector	getS3Connector(String name) Get a previously created S3 compliant storage connector from the feature store.
StorageConnector.SnowflakeConnector	getSnowflakeConnector(String name) Get a previously created Snowflake compliant storage connector from the feature store.
StorageConnector	getStorageConnector(String name) Get a previously created storage connector from the feature store.
abstract Object	getStreamFeatureGroup(String name) Get a feature group object with default version `1` from the feature store.
abstract Object	getStreamFeatureGroup(@NonNull String name, @NonNull Integer version) Get a feature group object from the feature store.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

featureGroupApi

protected FeatureGroupApi featureGroupApi

trainingDatasetApi

protected TrainingDatasetApi trainingDatasetApi

storageConnectorApi

protected StorageConnectorApi storageConnectorApi

LOGGER

protected static final org.slf4j.Logger LOGGER

DEFAULT_VERSION

protected static final Integer DEFAULT_VERSION

Constructor Detail

FeatureStoreBase

public FeatureStoreBase()

Method Detail

createStreamFeatureGroup

public abstract FeatureGroupBase createStreamFeatureGroup(@NonNull
                                                          @NonNull String name,
                                                          Integer version,
                                                          String description,
                                                          Boolean onlineEnabled,
                                                          TimeTravelFormat timeTravelFormat,
                                                          List<String> primaryKeys,
                                                          List<String> partitionKeys,
                                                          String eventTime,
                                                          String hudiPrecombineKey,
                                                          List<Feature> features,
                                                          StatisticsConfig statisticsConfig,
                                                          StorageConnector storageConnector,
                                                          String path)

Create a feature group object.

Parameters:

name - the name of the feature group

version - the version of the feature group

description - descrption of the feature group

onlineEnabled - whether the feature group should be online enabled

timeTravelFormat - the data format to use to store the offline data

primaryKeys - list of primary keys

partitionKeys - list of partition keys

eventTime - the feature/column to use as event time

hudiPrecombineKey - if the timeTravelFormat is set to hudi, the feature/column to use as precombine key

features - the list of feature objects if defined explicitly

statisticsConfig - the statistics configuration for the feature group

storageConnector - the storage connector to use to store the offline feature data (Default stored internally in Hopsworks).

path - the path on the storage where to store the feature data.

Returns:

The feature group metadata object.

getOrCreateStreamFeatureGroup

public abstract FeatureGroupBase getOrCreateStreamFeatureGroup(@NonNull
                                                               @NonNull String name,
                                                               Integer version,
                                                               String description,
                                                               Boolean onlineEnabled,
                                                               TimeTravelFormat timeTravelFormat,
                                                               List<String> primaryKeys,
                                                               List<String> partitionKeys,
                                                               String eventTime,
                                                               String hudiPrecombineKey,
                                                               List<Feature> features,
                                                               StatisticsConfig statisticsConfig,
                                                               StorageConnector storageConnector,
                                                               String path)
                                                        throws IOException,
                                                               FeatureStoreException

Get a feature group metadata object or create a new one if it doesn't exists.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StreamFeatureGroup streamFeatureGroup = featureStore.getOrCreateStreamFeatureGroup(
                 "doc_example",
                 1,
                 "Feature group for example in documentation",
                 true,
                 TimeTravelFormat.HUDI,
                 Collections.singletonList("primary_key"),
                 Collections.singletonList("partition_key"),
                 "event_time",
                 null,
                 features,
                 null,
                 new StatisticsConfig(true, true, true, true),
                 null
         );

         streamFeatureGroup.save()
 
 

Parameters:

name - the name of the feature group

version - the version of the feature group

description - descrption of the feature group

onlineEnabled - whether the feature group should be online enabled

timeTravelFormat - the data format to use to store the offline data

primaryKeys - list of primary keys

partitionKeys - list of partition keys

eventTime - the feature/column to use as event time

hudiPrecombineKey - if the timeTravelFormat is set to hudi, the feature/column to use as precombine key

features - the list of feature objects if defined explicitly

statisticsConfig - the statistics configuration for the feature group

storageConnector - the storage connector to use to store the offline feature data (Default stored internally in Hopsworks).

path - the path on the storage where to store the feature data.

Returns:

The feature group metadata object.

Throws:

IOException

FeatureStoreException

getStreamFeatureGroup

public abstract Object getStreamFeatureGroup(@NonNull
                                             @NonNull String name,
                                             @NonNull
                                             @NonNull Integer version)
                                      throws FeatureStoreException,
                                             IOException

Get a feature group object from the feature store.

Parameters:

name - the name of the feature group

version - the version of the feature group

Returns:

FeatureGroup

Throws:

FeatureStoreException

IOException

getStreamFeatureGroup

public abstract Object getStreamFeatureGroup(String name)
                                      throws FeatureStoreException,
                                             IOException

Get a feature group object with default version `1` from the feature store.

Parameters:

name - the name of the feature group

Returns:

FeatureGroup

Throws:

FeatureStoreException

IOException

getFeatureView

public abstract Object getFeatureView(String name)
                               throws FeatureStoreException,
                                      IOException

Throws:

FeatureStoreException

IOException

getFeatureView

public abstract Object getFeatureView(@NonNull
                                      @NonNull String name,
                                      @NonNull
                                      @NonNull Integer version)
                               throws FeatureStoreException,
                                      IOException

Throws:

FeatureStoreException

IOException

getStorageConnector

public StorageConnector getStorageConnector(String name)
                                     throws FeatureStoreException,
                                            IOException

Get a previously created storage connector from the feature store.

Storage connectors encapsulate all information needed for the execution engine to read and write to a specific storage.

If you want to connect to the online feature store, see the getOnlineStorageConnector` method to get the JDBC connector for the Online Feature Store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector sc = fs.getStorageConnector("sc_name");
 
 

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getHopsFsConnector

public StorageConnector.HopsFsConnector getHopsFsConnector(String name)
                                                    throws FeatureStoreException,
                                                           IOException

Get a previously created HopsFs compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.HopsFsConnector hfsSc = fs.getHopsFsConnector("hfs_sc_name");
 
 

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.HopsFsConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getJdbcConnector

public StorageConnector.JdbcConnector getJdbcConnector(String name)
                                                throws FeatureStoreException,
                                                       IOException

Get a previously created JDBC compliant storage connector from the feature store.

If you want to connect to the online feature store, see the getOnlineStorageConnector` method to get the JDBC connector for the Online Feature Store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.JdbcConnector jdbcSc = fs.getJdbcConnector("jdbc_sc_name");
 
 

Parameters:

name - Name of the jdbc storage connector to retrieve.

Returns:

StorageConnector.JdbcConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getOnlineStorageConnector

public StorageConnector.JdbcConnector getOnlineStorageConnector()
                                                         throws FeatureStoreException,
                                                                IOException

Get a previously created JDBC compliant storage connector from the feature store to connect to the online feature store.

 
        //get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.JdbcConnector onlineSc = fs.getOnlineStorageConnector("online_sc_name");
 
 

Returns:

StorageConnector.JdbcConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getS3Connector

public StorageConnector.S3Connector getS3Connector(String name)
                                            throws FeatureStoreException,
                                                   IOException

Get a previously created S3 compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.S3Connector s3Sc = fs.getS3Connector("s3_sc_name");
 
 

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.S3Connector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getRedshiftConnector

public StorageConnector.RedshiftConnector getRedshiftConnector(String name)
                                                        throws FeatureStoreException,
                                                               IOException

Get a previously created Redshift compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.RedshiftConnector rshSc = fs.getRedshiftConnector("rsh_sc_name");
 
 

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.RedshiftConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getSnowflakeConnector

public StorageConnector.SnowflakeConnector getSnowflakeConnector(String name)
                                                          throws FeatureStoreException,
                                                                 IOException

Get a previously created Snowflake compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.SnowflakeConnector snflSc = fs.getSnowflakeConnector("snfl_sc_name");
 
 

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.SnowflakeConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getAdlsConnector

public StorageConnector.AdlsConnector getAdlsConnector(String name)
                                                throws FeatureStoreException,
                                                       IOException

Get a previously created Adls compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.AdlsConnectorr adlslSc = fs.getAdlsConnector("adls_sc_name");
 
 

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.AdlsConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getKafkaConnector

public StorageConnector.KafkaConnector getKafkaConnector(String name)
                                                  throws FeatureStoreException,
                                                         IOException

Get a previously created Kafka compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.KafkaConnector kafkaSc = fs.getKafkaConnector("kafka_sc_name");
 
 

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.KafkaConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getBigqueryConnector

public StorageConnector.BigqueryConnector getBigqueryConnector(String name)
                                                        throws FeatureStoreException,
                                                               IOException

Get a previously created BigQuery compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.BigqueryConnector bigqSc = fs.getBigqueryConnector("bigq_sc_name");
 
 

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.BigqueryConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getGcsConnector

public StorageConnector.GcsConnector getGcsConnector(String name)
                                              throws FeatureStoreException,
                                                     IOException

Get a previously created Gcs compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.GcsConnector gcsSc = fs.getGcsConnector("gsc_sc_name");
 
 

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.GcsConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

toString

public String toString()

Overrides:

toString in class Object