com.logicalclocks.hsfs.spark

Class FeatureStore

java.lang.Object

com.logicalclocks.hsfs.FeatureStoreBase<Query>

com.logicalclocks.hsfs.spark.FeatureStore

public class FeatureStore
extends FeatureStoreBase<Query>

Field Summary

Fields inherited from class com.logicalclocks.hsfs.FeatureStoreBase

DEFAULT_VERSION, featureGroupApi, LOGGER, storageConnectorApi, trainingDatasetApi

Constructor Summary

Constructors

Constructor and Description
FeatureStore()

Method Summary

Modifier and Type	Method and Description
com.logicalclocks.hsfs.spark.ExternalFeatureGroup.ExternalFeatureGroupBuilder	createExternalFeatureGroup() Create external feature group builder object.
com.logicalclocks.hsfs.spark.FeatureGroup.FeatureGroupBuilder	createFeatureGroup() Create feature group builder object.
FeatureView.FeatureViewBuilder	createFeatureView()
com.logicalclocks.hsfs.spark.ExternalFeatureGroup.ExternalFeatureGroupBuilder	createOnDemandFeatureGroup() Deprecated.
com.logicalclocks.hsfs.spark.StreamFeatureGroup.StreamFeatureGroupBuilder	createStreamFeatureGroup() Create stream feature group builder object.
com.logicalclocks.hsfs.spark.TrainingDataset.TrainingDatasetBuilder	createTrainingDataset() Deprecated.
StorageConnector.AdlsConnector	getAdlsConnector(String name) Get a previously created Adls compliant storage connector from the feature store.
StorageConnector.BigqueryConnector	getBigqueryConnector(String name) Get a previously created BigQuery compliant storage connector from the feature store.
ExternalFeatureGroup	getExternalFeatureGroup(String name) Get an external feature group object with default version `1` from the feature store.
ExternalFeatureGroup	getExternalFeatureGroup(@NonNull String name, @NonNull Integer version) Get an external feature group object from the feature store.
scala.collection.Seq<ExternalFeatureGroup>	getExternalFeatureGroups(@NonNull String name) Get a list of all versions of an external feature group from the feature store.
FeatureGroup	getFeatureGroup(String name) Get a feature group object with default version `1` from the feature store.
FeatureGroup	getFeatureGroup(@NonNull String name, @NonNull Integer version) Get a feature group object from the feature store.
scala.collection.Seq<FeatureGroup>	getFeatureGroups(@NonNull String name) Get a list of all versions of a feature group from the feature store.
FeatureView	getFeatureView(String name) Get a feature view object with the default version `1` from the selected feature store.
FeatureView	getFeatureView(@NonNull String name, @NonNull Integer version) Get a feature view object from the selected feature store.
StorageConnector.GcsConnector	getGcsConnector(String name) Get a previously created Gcs compliant storage connector from the feature store.
StorageConnector.HopsFsConnector	getHopsFsConnector(String name) Get a previously created HopsFs compliant storage connector from the feature store.
StorageConnector.JdbcConnector	getJdbcConnector(String name) Get a previously created JDBC compliant storage connector from the feature store.
StorageConnector.KafkaConnector	getKafkaConnector(String name) Get a previously created Kafka compliant storage connector from the feature store.
ExternalFeatureGroup	getOnDemandFeatureGroup(String name) Deprecated.
ExternalFeatureGroup	getOnDemandFeatureGroup(@NonNull String name, @NonNull Integer version) Deprecated.
scala.collection.Seq<ExternalFeatureGroup>	getOnDemandFeatureGroups(@NonNull String name) Deprecated.
StorageConnector.JdbcConnector	getOnlineStorageConnector() Get a previously created JDBC compliant storage connector from the feature store to connect to the online feature store.
FeatureGroup	getOrCreateFeatureGroup(String name, Integer version) Get feature group metadata object or create a new one if it doesn't exist.
FeatureGroup	getOrCreateFeatureGroup(String name, Integer version, List<String> primaryKeys, boolean onlineEnabled, String eventTime) Get feature group metadata object or create a new one if it doesn't exist.
FeatureGroup	getOrCreateFeatureGroup(String name, Integer version, List<String> primaryKeys, List<String> partitionKeys, boolean onlineEnabled, String eventTime) Get feature group metadata object or create a new one if it doesn't exist.
FeatureGroup	getOrCreateFeatureGroup(String name, Integer version, String description, List<String> primaryKeys, List<String> partitionKeys, String hudiPrecombineKey, boolean onlineEnabled, TimeTravelFormat timeTravelFormat, StatisticsConfig statisticsConfig, String eventTime) Get feature group metadata object or create a new one if it doesn't exist.
FeatureView	getOrCreateFeatureView(String name, Query query, Integer version) Get feature view metadata object or create a new one if it doesn't exist.
FeatureView	getOrCreateFeatureView(String name, Query query, Integer version, String description, List<String> labels) Get feature view metadata object or create a new one if it doesn't exist.
StreamFeatureGroup	getOrCreateStreamFeatureGroup(String name, Integer version) Get stream feature group metadata object or create a new one if it doesn't exist.
StreamFeatureGroup	getOrCreateStreamFeatureGroup(String name, Integer version, List<String> primaryKeys, boolean onlineEnabled, String eventTime) Get stream feature group metadata object or create a new one if it doesn't exist.
StreamFeatureGroup	getOrCreateStreamFeatureGroup(String name, Integer version, List<String> primaryKeys, List<String> partitionKeys, boolean onlineEnabled, String eventTime) Get stream feature group metadata object or create a new one if it doesn't exist.
StreamFeatureGroup	getOrCreateStreamFeatureGroup(String name, Integer version, String description, List<String> primaryKeys, List<String> partitionKeys, String hudiPrecombineKey, boolean onlineEnabled, StatisticsConfig statisticsConfig, String eventTime) Get stream feature group metadata object or create a new one if it doesn't exist.
StorageConnector.RedshiftConnector	getRedshiftConnector(String name) Get a previously created Redshift compliant storage connector from the feature store.
StorageConnector.S3Connector	getS3Connector(String name) Get a previously created S3 compliant storage connector from the feature store.
StorageConnector.SnowflakeConnector	getSnowflakeConnector(String name) Get a previously created Snowflake compliant storage connector from the feature store.
StorageConnector	getStorageConnector(String name) Get a previously created storage connector from the feature store.
StreamFeatureGroup	getStreamFeatureGroup(String name) Get a stream feature group object with default version `1` from the feature store.
StreamFeatureGroup	getStreamFeatureGroup(@NonNull String name, @NonNull Integer version) Get a stream feature group object from the feature store.
TrainingDataset	getTrainingDataset(String name) Deprecated.
TrainingDataset	getTrainingDataset(@NonNull String name, @NonNull Integer version) Deprecated.
scala.collection.Seq<TrainingDataset>	getTrainingDatasets(@NonNull String name) Deprecated.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	sql(String query)

Methods inherited from class com.logicalclocks.hsfs.FeatureStoreBase

toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

FeatureStore

public FeatureStore()

Method Detail

getFeatureGroup

public FeatureGroup getFeatureGroup(@NonNull
                                    @NonNull String name,
                                    @NonNull
                                    @NonNull Integer version)
                             throws FeatureStoreException,
                                    IOException

Get a feature group object from the feature store.

Feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getFeatureGroup("electricity_prices", 1);
 
 

Parameters:

name - the name of the feature group

version - the version of the feature group

Returns:

FeatureGroup The feature group metadata object.

Throws:

FeatureStoreException - If unable to retrieve feature group from the feature store.

IOException - Generic IO exception.

getFeatureGroup

public FeatureGroup getFeatureGroup(String name)
                             throws FeatureStoreException,
                                    IOException

Get a feature group object with default version `1` from the feature store.

Feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getFeatureGroup("electricity_prices");
 
 

Parameters:

name - the name of the feature group

Returns:

FeatureGroup The feature group metadata object.

Throws:

FeatureStoreException - If unable to retrieve feature group from the feature store.

IOException - Generic IO exception.

getFeatureGroups

public scala.collection.Seq<FeatureGroup> getFeatureGroups(@NonNull
                                                           @NonNull String name)
                                                    throws FeatureStoreException,
                                                           IOException

Get a list of all versions of a feature group from the feature store.

Feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getFeatureGroups("electricity_prices");
 
 

Specified by:

getFeatureGroups in class FeatureStoreBase<Query>

Parameters:

name - the name of the feature group

Returns:

List of FeatureGroup metadata objects.

Throws:

FeatureStoreException - If unable to retrieve feature group from the feature store.

IOException - Generic IO exception.

createFeatureGroup

public com.logicalclocks.hsfs.spark.FeatureGroup.FeatureGroupBuilder createFeatureGroup()

Create feature group builder object.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup.FeatureGroupBuilder = fs.createFeatureGroup()
 
 

Specified by:

createFeatureGroup in class FeatureStoreBase<Query>

Returns:

FeatureGroup.FeatureGroupBuilder a FeatureGroup builder object.

getOrCreateFeatureGroup

public FeatureGroup getOrCreateFeatureGroup(String name,
                                            Integer version)
                                     throws IOException,
                                            FeatureStoreException

Get feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateFeatureGroup("fg_name", 1);
 
 

Specified by:

getOrCreateFeatureGroup in class FeatureStoreBase<Query>

Parameters:

name - of the feature group to retrieve or create.

version - of the feature group to retrieve or create.

Returns:

FeatureGroup The feature group metadata object.

Throws:

IOException - Generic IO exception.

FeatureStoreException - If unable to retrieve FeatureGroup from the feature store.

getOrCreateFeatureGroup

public FeatureGroup getOrCreateFeatureGroup(String name,
                                            Integer version,
                                            List<String> primaryKeys,
                                            boolean onlineEnabled,
                                            String eventTime)
                                     throws IOException,
                                            FeatureStoreException

Get feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateFeatureGroup("fg_name", 1, primaryKeys, true, "datetime");
 
 

Specified by:

getOrCreateFeatureGroup in class FeatureStoreBase<Query>

Parameters:

name - of the feature group to retrieve or create.

version - of the feature group to retrieve or create.

primaryKeys - A list of feature names to be used as primary key for the feature group. This primary key can be a composite key of multiple features and will be used as joining key.

onlineEnabled - Define whether the feature group should be made available also in the online feature store for low latency access.

eventTime - Name of the feature containing the event time for the features in this feature group. If eventTime is set the feature group can be used for point-in-time joins. The supported data types for the eventTime column are: timestamp, date and bigint

Returns:

FeatureGroup: The feature group metadata object

Throws:

IOException - Generic IO exception.

FeatureStoreException - If unable to retrieve FeatureGroup from the feature store.

getOrCreateFeatureGroup

public FeatureGroup getOrCreateFeatureGroup(String name,
                                            Integer version,
                                            List<String> primaryKeys,
                                            List<String> partitionKeys,
                                            boolean onlineEnabled,
                                            String eventTime)
                                     throws IOException,
                                            FeatureStoreException

Get feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateFeatureGroup("fg_name", 1, primaryKeys, partitionKeys, true, "datetime");
 
 

Specified by:

getOrCreateFeatureGroup in class FeatureStoreBase<Query>

Parameters:

name - of the feature group to retrieve or create.

version - of the feature group to retrieve or create.

primaryKeys - A list of feature names to be used as primary key for the feature group. This primary key can be a composite key of multiple features and will be used as joining key.

partitionKeys - A list of feature names to be used as partition key when writing the feature data to the offline storage.

onlineEnabled - Define whether the feature group should be made available also in the online feature store for low latency access.

eventTime - Name of the feature containing the event time for the features in this feature group. If eventTime is set the feature group can be used for point-in-time joins. The supported data types for the eventTime column are: timestamp, date and bigint

Returns:

FeatureGroup: The feature group metadata object

Throws:

IOException - Generic IO exception.

FeatureStoreException - If unable to retrieve FeatureGroup from the feature store.

getOrCreateFeatureGroup

public FeatureGroup getOrCreateFeatureGroup(String name,
                                            Integer version,
                                            String description,
                                            List<String> primaryKeys,
                                            List<String> partitionKeys,
                                            String hudiPrecombineKey,
                                            boolean onlineEnabled,
                                            TimeTravelFormat timeTravelFormat,
                                            StatisticsConfig statisticsConfig,
                                            String eventTime)
                                     throws IOException,
                                            FeatureStoreException

Get feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateFeatureGroup("fg_name", 1, primaryKeys, partitionKeys, true, "datetime");
 
 

Specified by:

getOrCreateFeatureGroup in class FeatureStoreBase<Query>

Parameters:

name - of the feature group to retrieve or create.

version - of the feature group to retrieve or create.

description - contents of the feature group to improve discoverability for Data Scientists

primaryKeys - A list of feature names to be used as primary key for the feature group. This primary key can be a composite key of multiple features and will be used as joining key.

partitionKeys - A list of feature names to be used as partition key when writing the feature data to the offline storage.

hudiPrecombineKey - A feature name to be used as a precombine key for the `HUDI` feature group. If feature group has time travel format `HUDI` and hudi precombine key was not specified then the first primary key of the feature group will be used as hudi precombine key.

onlineEnabled - Define whether the feature group should be made available also in the online feature store for low latency access.

timeTravelFormat - Format used for time travel: `TimeTravelFormat.HUDI` or `TimeTravelFormat.NONE`.

statisticsConfig - A configuration object, to generally enable descriptive statistics computation for this feature group, `"correlations`" to turn on feature correlation computation, `"histograms"` to compute feature value frequencies and `"exact_uniqueness"` to compute uniqueness, distinctness and entropy. The values should be booleans indicating the setting. To fully turn off statistics computation pass `statisticsConfig=false`.

eventTime - Name of the feature containing the event time for the features in this feature group. If eventTime is set the feature group can be used for point-in-time joins. The supported data types for the eventTime column are: timestamp, date and bigint

Returns:

FeatureGroup: The feature group metadata object.

Throws:

IOException - Generic IO exception.

FeatureStoreException - If unable to retrieve FeatureGroup from the feature store.

getStreamFeatureGroup

public StreamFeatureGroup getStreamFeatureGroup(String name)
                                         throws FeatureStoreException,
                                                IOException

Get a stream feature group object with default version `1` from the feature store.

Getting a stream feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getStreamFeatureGroup("electricity_prices");
 <
 

Specified by:

getStreamFeatureGroup in class FeatureStoreBase<Query>

Parameters:

name - the name of the feature group

Returns:

FeatureGroup

Throws:

FeatureStoreException - If unable to retrieve feature group from the feature store.

IOException - Generic IO exception.

getStreamFeatureGroup

public StreamFeatureGroup getStreamFeatureGroup(@NonNull
                                                @NonNull String name,
                                                @NonNull
                                                @NonNull Integer version)
                                         throws FeatureStoreException,
                                                IOException

Get a stream feature group object from the feature store.

Getting a stream feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getStreamFeatureGroup("electricity_prices", 1);
 
 

Specified by:

getStreamFeatureGroup in class FeatureStoreBase<Query>

Parameters:

name - the name of the feature group

version - the version of the feature group

Returns:

StreamFeatureGroup The stream feature group metadata object.

Throws:

FeatureStoreException - If unable to retrieve feature group from the feature store.

IOException - Generic IO exception.

createStreamFeatureGroup

public com.logicalclocks.hsfs.spark.StreamFeatureGroup.StreamFeatureGroupBuilder createStreamFeatureGroup()

Create stream feature group builder object.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StreamFeatureGroup.StreamFeatureGroupBuilder = fs.createStreamFeatureGroup()
 
 

Specified by:

createStreamFeatureGroup in class FeatureStoreBase<Query>

Returns:

StreamFeatureGroup.StreamFeatureGroupBuilder a StreamFeatureGroup builder object.

getOrCreateStreamFeatureGroup

public StreamFeatureGroup getOrCreateStreamFeatureGroup(String name,
                                                        Integer version)
                                                 throws IOException,
                                                        FeatureStoreException

Get stream feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateStreamFeatureGroup("fg_name", 1);
 
 

Specified by:

getOrCreateStreamFeatureGroup in class FeatureStoreBase<Query>

Parameters:

name - of the feature group to retrieve or create.

version - of the feature group to retrieve or create.

Returns:

FeatureGroup The feature group metadata object.

Throws:

IOException - Generic IO exception.

FeatureStoreException - If unable to retrieve StreamFeatureGroup from the feature store.

getOrCreateStreamFeatureGroup

public StreamFeatureGroup getOrCreateStreamFeatureGroup(String name,
                                                        Integer version,
                                                        List<String> primaryKeys,
                                                        boolean onlineEnabled,
                                                        String eventTime)
                                                 throws IOException,
                                                        FeatureStoreException

Get stream feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateStreamFeatureGroup("fg_name", 1, primaryKeys, true, "datetime");
 
 

Specified by:

getOrCreateStreamFeatureGroup in class FeatureStoreBase<Query>

Parameters:

name - Name of the feature group to retrieve or create.

version - Version of the feature group to retrieve or create.

primaryKeys - A list of feature names to be used as primary key for the feature group. This primary key can be a composite key of multiple features and will be used as joining key.

onlineEnabled - Define whether the feature group should be made available also in the online feature store for low latency access.

eventTime - Name of the feature containing the event time for the features in this feature group. If eventTime is set the feature group can be used for point-in-time joins. The supported data types for the eventTime column are: timestamp, date and bigint

Returns:

FeatureGroup The feature group metadata object.

Throws:

IOException - Generic IO exception.

FeatureStoreException - If unable to retrieve StreamFeatureGroup from the feature store.

getOrCreateStreamFeatureGroup

public StreamFeatureGroup getOrCreateStreamFeatureGroup(String name,
                                                        Integer version,
                                                        List<String> primaryKeys,
                                                        List<String> partitionKeys,
                                                        boolean onlineEnabled,
                                                        String eventTime)
                                                 throws IOException,
                                                        FeatureStoreException

Get stream feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getOrCreateStreamFeatureGroup("fg_name", 1, primaryKeys, partitionKeys, true,
        "datetime");
 
 

Specified by:

getOrCreateStreamFeatureGroup in class FeatureStoreBase<Query>

Parameters:

name - Name of the feature group to retrieve or create.

version - Version of the feature group to retrieve or create.

primaryKeys - A list of feature names to be used as primary key for the feature group. This primary key can be a composite key of multiple features and will be used as joining key.

partitionKeys - A list of feature names to be used as partition key when writing the feature data to the offline storage.

onlineEnabled - Define whether the feature group should be made available also in the online feature store for low latency access.

eventTime - Name of the feature containing the event time for the features in this feature group. If eventTime is set the feature group can be used for point-in-time joins. The supported data types for the eventTime column are: timestamp, date and bigint

Returns:

FeatureGroup: The feature group metadata object

Throws:

IOException - Generic IO exception.

FeatureStoreException - If unable to retrieve StreamFeatureGroup from the feature store.

getOrCreateStreamFeatureGroup

public StreamFeatureGroup getOrCreateStreamFeatureGroup(String name,
                                                        Integer version,
                                                        String description,
                                                        List<String> primaryKeys,
                                                        List<String> partitionKeys,
                                                        String hudiPrecombineKey,
                                                        boolean onlineEnabled,
                                                        StatisticsConfig statisticsConfig,
                                                        String eventTime)
                                                 throws IOException,
                                                        FeatureStoreException

Get stream feature group metadata object or create a new one if it doesn't exist. This method doesn't update existing feature group metadata.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StreamFeatureGroup fg = fs.getOrCreateStreamFeatureGroup("fg_name", 1, primaryKeys,
        partitionKeys, true, "datetime");
 
 

Specified by:

getOrCreateStreamFeatureGroup in class FeatureStoreBase<Query>

Parameters:

name - of the feature group to retrieve or create.

version - of the feature group to retrieve or create.

description - contents of the feature group to improve discoverability for Data Scientists

primaryKeys - A list of feature names to be used as primary key for the feature group. This primary key can be a composite key of multiple features and will be used as joining key.

partitionKeys - A list of feature names to be used as partition key when writing the feature data to the offline storage.

hudiPrecombineKey - A feature name to be used as a precombine key for the `HUDI` feature group. If feature group has time travel format `HUDI` and hudi precombine key was not specified then the first primary key of the feature group will be used as hudi precombine key.

onlineEnabled - Define whether the feature group should be made available also in the online feature store for low latency access.

statisticsConfig - A configuration object, to generally enable descriptive statistics computation for this feature group, `"correlations`" to turn on feature correlation computation, `"histograms"` to compute feature value frequencies and `"exact_uniqueness"` to compute uniqueness, distinctness and entropy. The values should be booleans indicating the setting. To fully turn off statistics computation pass `statisticsConfig=false`.

eventTime - Name of the feature containing the event time for the features in this feature group. If eventTime is set the feature group can be used for point-in-time joins.

Returns:

FeatureGroup: The feature group metadata object.

Throws:

IOException - Generic IO exception.

FeatureStoreException - If unable to retrieve FeatureGroup from the feature store.

createExternalFeatureGroup

public com.logicalclocks.hsfs.spark.ExternalFeatureGroup.ExternalFeatureGroupBuilder createExternalFeatureGroup()

Create external feature group builder object.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        ExternalFeatureGroup.ExternalFeatureGroupBuilder = fs.createExternalFeatureGroup()
 
 

Specified by:

createExternalFeatureGroup in class FeatureStoreBase<Query>

Returns:

ExternalFeatureGroup.ExternalFeatureGroupBuilder a ExternalFeatureGroup builder object.

createOnDemandFeatureGroup

@Deprecated
public com.logicalclocks.hsfs.spark.ExternalFeatureGroup.ExternalFeatureGroupBuilder createOnDemandFeatureGroup()

Deprecated.

getExternalFeatureGroups

public scala.collection.Seq<ExternalFeatureGroup> getExternalFeatureGroups(@NonNull
                                                                           @NonNull String name)
                                                                    throws FeatureStoreException,
                                                                           IOException

Get a list of all versions of an external feature group from the feature store.

Feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureGroup fg = fs.getExternalFeatureGroups("external_fg_name");
 
 

Specified by:

getExternalFeatureGroups in class FeatureStoreBase<Query>

Parameters:

name - The name of the feature group.

Returns:

List of ExternalFeatureGroup metadata objects.

Throws:

FeatureStoreException - If unable to retrieve feature group from the feature store.

IOException - Generic IO exception.

getExternalFeatureGroup

public ExternalFeatureGroup getExternalFeatureGroup(@NonNull
                                                    @NonNull String name,
                                                    @NonNull
                                                    @NonNull Integer version)
                                             throws FeatureStoreException,
                                                    IOException

Get an external feature group object from the feature store.

Getting a stream feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        ExternalFeatureGroup fg = fs.getExternalFeatureGroup("external_fg_name", 1);
 
 

Specified by:

getExternalFeatureGroup in class FeatureStoreBase<Query>

Parameters:

name - the name of the feature group

version - the version of the feature group

Returns:

ExternalFeatureGroup The external feature group metadata object.

Throws:

FeatureStoreException - If unable to retrieve feature group from the feature store.

IOException - Generic IO exception.

getExternalFeatureGroup

public ExternalFeatureGroup getExternalFeatureGroup(String name)
                                             throws FeatureStoreException,
                                                    IOException

Get an external feature group object with default version `1` from the feature store.

Getting external feature group metadata handle enables to interact with the feature group, such as read the data or use the `Query`-API to perform joins between feature groups and create feature views.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        ExternalFeatureGroup fg = fs.getExternalFeatureGroup("external_fg_name");
 
 

Specified by:

getExternalFeatureGroup in class FeatureStoreBase<Query>

Parameters:

name - the name of the feature group

Returns:

ExternalFeatureGroup The external feature group metadata object.

Throws:

FeatureStoreException - If unable to retrieve feature group from the feature store.

IOException - Generic IO exception.

getStorageConnector

public StorageConnector getStorageConnector(String name)
                                     throws FeatureStoreException,
                                            IOException

Get a previously created storage connector from the feature store.

Storage connectors encapsulate all information needed for the execution engine to read and write to a specific storage.

If you want to connect to the online feature store, see the getOnlineStorageConnector` method to get the JDBC connector for the Online Feature Store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector sc = fs.getStorageConnector("sc_name");
 
 

Specified by:

getStorageConnector in class FeatureStoreBase<Query>

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getHopsFsConnector

public StorageConnector.HopsFsConnector getHopsFsConnector(String name)
                                                    throws FeatureStoreException,
                                                           IOException

Get a previously created HopsFs compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.HopsFsConnector hfsSc = fs.getHopsFsConnector("hfs_sc_name");
 
 

Specified by:

getHopsFsConnector in class FeatureStoreBase<Query>

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.HopsFsConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getJdbcConnector

public StorageConnector.JdbcConnector getJdbcConnector(String name)
                                                throws FeatureStoreException,
                                                       IOException

Get a previously created JDBC compliant storage connector from the feature store.

If you want to connect to the online feature store, see the getOnlineStorageConnector` method to get the JDBC connector for the Online Feature Store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.JdbcConnector jdbcSc = fs.getJdbcConnector("jdbc_sc_name");
 
 

Specified by:

getJdbcConnector in class FeatureStoreBase<Query>

Parameters:

name - Name of the jdbc storage connector to retrieve.

Returns:

StorageConnector.JdbcConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getOnlineStorageConnector

public StorageConnector.JdbcConnector getOnlineStorageConnector()
                                                         throws FeatureStoreException,
                                                                IOException

Get a previously created JDBC compliant storage connector from the feature store to connect to the online feature store.

 
        //get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.JdbcConnector onlineSc = fs.getOnlineStorageConnector("online_sc_name");
 
 

Specified by:

getOnlineStorageConnector in class FeatureStoreBase<Query>

Returns:

StorageConnector.JdbcConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getS3Connector

public StorageConnector.S3Connector getS3Connector(String name)
                                            throws FeatureStoreException,
                                                   IOException

Get a previously created S3 compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.S3Connector s3Sc = fs.getS3Connector("s3_sc_name");
 
 

Specified by:

getS3Connector in class FeatureStoreBase<Query>

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.S3Connector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getRedshiftConnector

public StorageConnector.RedshiftConnector getRedshiftConnector(String name)
                                                        throws FeatureStoreException,
                                                               IOException

Get a previously created Redshift compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.RedshiftConnector rshSc = fs.getRedshiftConnector("rsh_sc_name");
 
 

Specified by:

getRedshiftConnector in class FeatureStoreBase<Query>

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.RedshiftConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getSnowflakeConnector

public StorageConnector.SnowflakeConnector getSnowflakeConnector(String name)
                                                          throws FeatureStoreException,
                                                                 IOException

Get a previously created Snowflake compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.SnowflakeConnector snflSc = fs.getSnowflakeConnector("snfl_sc_name");
 
 

Specified by:

getSnowflakeConnector in class FeatureStoreBase<Query>

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.SnowflakeConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getAdlsConnector

public StorageConnector.AdlsConnector getAdlsConnector(String name)
                                                throws FeatureStoreException,
                                                       IOException

Get a previously created Adls compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.AdlsConnectorr adlslSc = fs.getAdlsConnector("adls_sc_name");
 
 

Specified by:

getAdlsConnector in class FeatureStoreBase<Query>

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.AdlsConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getKafkaConnector

public StorageConnector.KafkaConnector getKafkaConnector(String name)
                                                  throws FeatureStoreException,
                                                         IOException

Get a previously created Kafka compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.KafkaConnector kafkaSc = fs.getKafkaConnector("kafka_sc_name");
 
 

Specified by:

getKafkaConnector in class FeatureStoreBase<Query>

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.KafkaConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getBigqueryConnector

public StorageConnector.BigqueryConnector getBigqueryConnector(String name)
                                                        throws FeatureStoreException,
                                                               IOException

Get a previously created BigQuery compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.BigqueryConnector bigqSc = fs.getBigqueryConnector("bigq_sc_name");
 
 

Specified by:

getBigqueryConnector in class FeatureStoreBase<Query>

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.BigqueryConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getGcsConnector

public StorageConnector.GcsConnector getGcsConnector(String name)
                                              throws FeatureStoreException,
                                                     IOException

Get a previously created Gcs compliant storage connector from the feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        StorageConnector.GcsConnector gcsSc = fs.getGcsConnector("gsc_sc_name");
 
 

Specified by:

getGcsConnector in class FeatureStoreBase<Query>

Parameters:

name - Name of the storage connector to retrieve.

Returns:

StorageConnector.GcsConnector Storage connector object.

Throws:

FeatureStoreException - If unable to retrieve StorageConnector from the feature store.

IOException - Generic IO exception.

getOnDemandFeatureGroup

@Deprecated
public ExternalFeatureGroup getOnDemandFeatureGroup(@NonNull
                                                                @NonNull String name,
                                                                @NonNull
                                                                @NonNull Integer version)
                                                         throws FeatureStoreException,
                                                                IOException

Deprecated.

Throws:

FeatureStoreException

IOException

getOnDemandFeatureGroup

@Deprecated
public ExternalFeatureGroup getOnDemandFeatureGroup(String name)
                                                         throws FeatureStoreException,
                                                                IOException

Deprecated.

Throws:

FeatureStoreException

IOException

getOnDemandFeatureGroups

@Deprecated
public scala.collection.Seq<ExternalFeatureGroup> getOnDemandFeatureGroups(@NonNull
                                                                                       @NonNull String name)
                                                                                throws FeatureStoreException,
                                                                                       IOException

Deprecated.

Throws:

FeatureStoreException

IOException

createFeatureView

public FeatureView.FeatureViewBuilder createFeatureView()

Specified by:

createFeatureView in class FeatureStoreBase<Query>

getOrCreateFeatureView

public FeatureView getOrCreateFeatureView(String name,
                                          Query query,
                                          Integer version)
                                   throws FeatureStoreException,
                                          IOException

Get feature view metadata object or create a new one if it doesn't exist. This method doesn't update existing feature view metadata.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureView fv = fs.getOrCreateFeatureView("fv_name", query, 1);
 
 

Specified by:

getOrCreateFeatureView in class FeatureStoreBase<Query>

Parameters:

name - Name of the feature view.

query - Query object.

version - Version of the feature view.

Returns:

FeatureView The feature view metadata object.

Throws:

FeatureStoreException - If unable to retrieve FeatureView from the feature store.

IOException - Generic IO exception.

getOrCreateFeatureView

public FeatureView getOrCreateFeatureView(String name,
                                          Query query,
                                          Integer version,
                                          String description,
                                          List<String> labels)
                                   throws FeatureStoreException,
                                          IOException

Get feature view metadata object or create a new one if it doesn't exist. This method doesn't update existing feature view metadata.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureView fv = fs.getOrCreateFeatureView("fv_name", query, 1, description, labels);
        "datetime");
 
 

Specified by:

getOrCreateFeatureView in class FeatureStoreBase<Query>

Parameters:

name - Name of the feature view.

query - Query object.

version - Version of the feature view.

description - Description of the feature view.

labels - List of label features.

Returns:

FeatureView The feature view metadata object.

Throws:

FeatureStoreException - If unable to retrieve FeatureView from the feature store.

IOException - Generic IO exception.

getFeatureView

public FeatureView getFeatureView(@NonNull
                                  @NonNull String name,
                                  @NonNull
                                  @NonNull Integer version)
                           throws FeatureStoreException,
                                  IOException

Get a feature view object from the selected feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureView fv = fs.getFeatureView("fv_name", 1);
 
 

Specified by:

getFeatureView in class FeatureStoreBase<Query>

Parameters:

name - Name of the feature view.

version - Version to get.

Returns:

FeatureView The feature view metadata object.

Throws:

FeatureStoreException - If unable to retrieve FeatureView from the feature store.

IOException - Generic IO exception.

getFeatureView

public FeatureView getFeatureView(String name)
                           throws FeatureStoreException,
                                  IOException

Get a feature view object with the default version `1` from the selected feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        FeatureView fv = fs.getFeatureView("fv_name");
 
 

Specified by:

getFeatureView in class FeatureStoreBase<Query>

Parameters:

name - Name of the feature view.

Returns:

FeatureView The feature view metadata object.

Throws:

FeatureStoreException - If unable to retrieve FeatureView from the feature store.

IOException - Generic IO exception.

sql

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> sql(String query)

Specified by:

sql in class FeatureStoreBase<Query>

createTrainingDataset

@Deprecated
public com.logicalclocks.hsfs.spark.TrainingDataset.TrainingDatasetBuilder createTrainingDataset()

Deprecated.

getTrainingDataset

@Deprecated
public TrainingDataset getTrainingDataset(@NonNull
                                                      @NonNull String name,
                                                      @NonNull
                                                      @NonNull Integer version)
                                               throws FeatureStoreException,
                                                      IOException

Deprecated.

Description copied from class: FeatureStoreBase

Get a training dataset object from the selected feature store.

Specified by:

getTrainingDataset in class FeatureStoreBase<Query>

Parameters:

name - name of the training dataset

version - version to get

Returns:

TrainingDataset

Throws:

FeatureStoreException

IOException

getTrainingDataset

@Deprecated
public TrainingDataset getTrainingDataset(String name)
                                               throws FeatureStoreException,
                                                      IOException

Deprecated.

Description copied from class: FeatureStoreBase

Get a training dataset object with the default version `1` from the selected feature store.

Specified by:

getTrainingDataset in class FeatureStoreBase<Query>

Parameters:

name - name of the training dataset

Returns:

TrainingDataset

Throws:

FeatureStoreException

IOException

getTrainingDatasets

@Deprecated
public scala.collection.Seq<TrainingDataset> getTrainingDatasets(@NonNull
                                                                             @NonNull String name)
                                                                      throws FeatureStoreException,
                                                                             IOException

Deprecated.

Description copied from class: FeatureStoreBase

Get all versions of a training dataset object from the selected feature store.

Specified by:

getTrainingDatasets in class FeatureStoreBase<Query>

Parameters:

name - name of the training dataset

Returns:

TrainingDataset

Throws:

FeatureStoreException

IOException