com.logicalclocks.hsfs

Class FeatureViewBase<T extends FeatureViewBase,T3 extends FeatureStoreBase<T4>,T4 extends QueryBase,T5>

java.lang.Object

com.logicalclocks.hsfs.FeatureViewBase<T,T3,T4,T5>

Direct Known Subclasses:

FeatureView, FeatureView, FeatureView, FeatureView

public abstract class FeatureViewBase<T extends FeatureViewBase,T3 extends FeatureStoreBase<T4>,T4 extends QueryBase,T5>
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
protected String	description
protected Integer	extraFilterVersion
protected List<TrainingDatasetFeature>	features
protected T3	featureStore
protected FeatureViewApi	featureViewApi
protected Integer	id
protected List<String>	labels
protected static org.slf4j.Logger	LOGGER
protected String	name
protected T4	query
protected TagsApi	tagsApi
protected String	type
protected VectorServer	vectorServer
protected Integer	version

Constructor Summary

Constructors

Constructor and Description
FeatureViewBase()

Method Summary

Modifier and Type	Method and Description
void	addTag(String name, Object value) Add name/value tag to the feature view.
void	addTrainingDatasetTag(Integer version, String name, Object value) Add name/value tag to the training dataset.
void	closeVectorServer() Closes the ExecutorService and JDBC DataSource used to retrieve feature vectors from the online feature store.
Integer	createTrainingData(String startTime, String endTime, String description, DataFormat dataFormat) Create the metadata for a training dataset and save the corresponding training data into `location`.
Integer	createTrainingData(String startTime, String endTime, String description, DataFormat dataFormat, Boolean coalesce, StorageConnector storageConnector, String location, Long seed, StatisticsConfig statisticsConfig, Map<String,String> writeOptions, FilterLogic extraFilterLogic, Filter extraFilter) Create the metadata for a training dataset and save the corresponding training data into `location`.
Integer	createTrainTestSplit(Float testSize, String trainStart, String trainEnd, String testStart, String testEnd, String description, DataFormat dataFormat) Create the metadata for a training dataset and save the corresponding training data into `location`.
Integer	createTrainTestSplit(Float testSize, String trainStart, String trainEnd, String testStart, String testEnd, String description, DataFormat dataFormat, Boolean coalesce, StorageConnector storageConnector, String location, Long seed, StatisticsConfig statisticsConfig, Map<String,String> writeOptions, FilterLogic extraFilterLogic, Filter extraFilter) Create the metadata for a training dataset and save the corresponding training data into `location`.
Integer	createTrainValidationTestSplit(Float validationSize, Float testSize, String trainStart, String trainEnd, String validationStart, String validationEnd, String testStart, String testEnd, String description, DataFormat dataFormat) Create the metadata for a training dataset and save the corresponding training data into `location`.
Integer	createTrainValidationTestSplit(Float validationSize, Float testSize, String trainStart, String trainEnd, String validationStart, String validationEnd, String testStart, String testEnd, String description, DataFormat dataFormat, Boolean coalesce, StorageConnector storageConnector, String location, Long seed, StatisticsConfig statisticsConfig, Map<String,String> writeOptions, FilterLogic extraFilterLogic, Filter extraFilter) Create the metadata for a training dataset and save the corresponding training data into `location`.
void	delete() Delete current feature view, all associated metadata and training data.
void	deleteTag(String name) Delete a tag of the feature view.
void	deleteTrainingDatasetTag(Integer version, String name) Delete a tag of the training dataset.
List<Object>	getFeatureVector(Map<String,Object> entry) Returns assembled feature vector from online feature store.
List<Object>	getFeatureVector(Map<String,Object> entry, boolean external) Returns assembled feature vector from online feature store.
List<List<Object>>	getFeatureVectors(Map<String,List<Object>> entry) Returns assembled feature vectors in batches from online feature store.
List<List<Object>>	getFeatureVectors(Map<String,List<Object>> entry, boolean external) Returns assembled feature vectors in batches from online feature store.
HashSet<String>	getPrimaryKeys() Set of primary key names that is used as keys in input dict object for `get_serving_vector` method.
Object	getTag(String name) Get a single tag value of the feature view.
Map<String,Object>	getTags() Get all tags of the feature view.
Object	getTrainingDatasetTag(Integer version, String name) Get a single tag value of the training dataset.
Map<String,Object>	getTrainingDatasetTags(Integer version) Get all tags of the training dataset.
void	initBatchScoring(Integer trainingDatasetVersion) Initialise feature view to retrieve feature vector from offline feature store.
void	initServing() Initialise feature view to retrieve feature vector from online feature store.
void	initServing(Boolean batch, Boolean external) Initialise feature view to retrieve feature vector from online feature store.
protected void	validateTrainTestSplit(Float testSize, String trainEnd, String testStart)
protected void	validateTrainValidationTestSplit(Float validationSize, Float testSize, String trainEnd, String validationStart, String validationEnd, String testStart)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

featureStore

protected T3 extends FeatureStoreBase<T4> featureStore

id

protected Integer id

name

protected String name

version

protected Integer version

description

protected String description

features

protected List<TrainingDatasetFeature> features

query

protected T4 extends QueryBase query

labels

protected List<String> labels

type

protected String type

LOGGER

protected static final org.slf4j.Logger LOGGER

featureViewApi

protected FeatureViewApi featureViewApi

tagsApi

protected TagsApi tagsApi

vectorServer

protected VectorServer vectorServer

extraFilterVersion

protected Integer extraFilterVersion

Constructor Detail

FeatureViewBase

public FeatureViewBase()

Method Detail

createTrainingData

public Integer createTrainingData(String startTime,
                                  String endTime,
                                  String description,
                                  DataFormat dataFormat)
                           throws IOException,
                                  FeatureStoreException,
                                  ParseException

Create the metadata for a training dataset and save the corresponding training data into `location`. The training data can be retrieved by calling `feature_view.getTrainingData()`.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // create training dataset
        String startTime = "20220101000000";
        String endTime = "20220606235959";
        String description = "demo training dataset":
        fv.createTrainingData(startTime, endTime, description, DataFormat.CSV);
 
 

Parameters:

startTime - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

endTime - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

description - A string describing the contents of the training dataset to improve discoverability for Data Scientists.

dataFormat - The data format used to save the training dataset.

Returns:

Integer Training dataset version.

Throws:

FeatureStoreException - If Client is not connected to Hopsworks and/or unable to identify format of the provided `startTime`/`endTime` date formats.

IOException - Generic IO exception.

ParseException - In case it's unable to parse provided `startTime`/`endTime` strings to date types.

createTrainingData

public Integer createTrainingData(String startTime,
                                  String endTime,
                                  String description,
                                  DataFormat dataFormat,
                                  Boolean coalesce,
                                  StorageConnector storageConnector,
                                  String location,
                                  Long seed,
                                  StatisticsConfig statisticsConfig,
                                  Map<String,String> writeOptions,
                                  FilterLogic extraFilterLogic,
                                  Filter extraFilter)
                           throws IOException,
                                  FeatureStoreException,
                                  ParseException

Create the metadata for a training dataset and save the corresponding training data into `location`. The training data can be retrieved by calling `featureView.getTrainingData()`.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // create training dataset
        String startTime = "20220101000000";
        String endTime = "20220606235959";
        String description = "demo training dataset":
        String location = "";
        StatisticsConfig statisticsConfig = new StatisticsConfig(true, true, true, true)
        fv.createTrainingData(startTime, endTime, description, DataFormat.CSV, true, location, statisticsConfig);
 
 

Parameters:

startTime - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

endTime - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

description - A string describing the contents of the training dataset to improve discoverability for Data Scientists.

dataFormat - The data format used to save the training dataset.

coalesce - If true the training dataset data will be coalesced into a single partition before writing. The resulting training dataset will be a single file per split.

storageConnector - Storage connector defining the sink location for the training dataset. If `null` is provided and materializes training dataset on HopsFS.

location - Path to complement the sink storage connector with, e.g if the storage connector points to an S3 bucket, this path can be used to define a sub-directory inside the bucket to place the training dataset. If empty string is provided `""`, saving the training dataset at the root defined by the storage connector.

seed - Define a seed to create the random splits with, in order to guarantee reproducability,

statisticsConfig - A configuration object, to generally enable descriptive statistics computation for this feature group, `"correlations`" to turn on feature correlation computation, `"histograms"` to compute feature value frequencies and `"exact_uniqueness"` to compute uniqueness, distinctness and entropy. The values should be booleans indicating the setting. To fully turn off statistics computation pass `statisticsConfig=null`.

writeOptions - Additional write options as key-value pairs.

extraFilterLogic - Additional filters (set of Filter objects) to be attached to the training dataset. The filters will be also applied in `getBatchData`.

extraFilter - Additional filter to be attached to the training dataset. The filter will be also applied in `getBatchData`.

Returns:

Integer Training dataset version.

Throws:

FeatureStoreException - If Client is not connected to Hopsworks and/or unable to identify format of the provided `startTime`/`endTime` date formats.

IOException - Generic IO exception.

ParseException - In case it's unable to parse provided `startTime`/`endTime` strings to date types.

createTrainTestSplit

public Integer createTrainTestSplit(Float testSize,
                                    String trainStart,
                                    String trainEnd,
                                    String testStart,
                                    String testEnd,
                                    String description,
                                    DataFormat dataFormat)
                             throws IOException,
                                    FeatureStoreException,
                                    ParseException

Create the metadata for a training dataset and save the corresponding training data into `location`. The training data is split into train and test set at random or according to time ranges. The training data can be retrieved by calling `featureView.getTrainTestSplit` method.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // create training dataset based on time split
        String trainStart = "20220101000000";
        String trainEnd = "20220630235959";
        String testStart = "20220701000000";
        String testEnd = "20220830235959";
        String description = "demo training dataset":
        fv.createTrainTestSplit(null, trainStart, trainEnd, testStart, testEnd, description, DataFormat.CSV);

        // or based on random split
        fv.createTrainTestSplit(30, null, null, null, null, description, DataFormat.CSV);
 
 

Parameters:

testSize - Size of test set.

trainStart - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

trainEnd - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

testStart - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

testEnd - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

description - A string describing the contents of the training dataset to improve discoverability for Data Scientists.

dataFormat - The data format used to save the training dataset.

Returns:

Integer Training dataset version

Throws:

FeatureStoreException - If Client is not connected to Hopsworks and/or unable to identify format of the provided date strings to date formats.

IOException - Generic IO exception.

ParseException - In case it's unable to parse provided date strings to date types.

createTrainTestSplit

public Integer createTrainTestSplit(Float testSize,
                                    String trainStart,
                                    String trainEnd,
                                    String testStart,
                                    String testEnd,
                                    String description,
                                    DataFormat dataFormat,
                                    Boolean coalesce,
                                    StorageConnector storageConnector,
                                    String location,
                                    Long seed,
                                    StatisticsConfig statisticsConfig,
                                    Map<String,String> writeOptions,
                                    FilterLogic extraFilterLogic,
                                    Filter extraFilter)
                             throws IOException,
                                    FeatureStoreException,
                                    ParseException

Create the metadata for a training dataset and save the corresponding training data into `location`. The training data is split into train and test set at random or according to time ranges. The training data can be retrieved by calling `featureView.getTrainTestSplit` method.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // create training dataset based on time split
        String trainStart = "20220101000000";
        String trainEnd = "20220630235959";
        String testStart = "20220701000000";
        String testEnd = "20220830235959";
        String description = "demo training dataset":
        StatisticsConfig statisticsConfig = new StatisticsConfig(true, true, true, true)
        Map<String, String> writeOptions = new HashMap<String, String>() {{
                           put("header", "true");
                           put("delimiter", ",")}
                           };
        // define extra filters
        Filter leftFtFilter = new Filter();
        leftFtFilter.setFeature(new Feature("left_ft_name"));
        leftFtFilter.setValue("400");
        leftFtFilter.setCondition(SqlFilterCondition.EQUALS);
        Filter rightFtFilter = new Filter();
        rightFtFilter.setFeature(new Feature("right_ft_name"));
        rightFtFilter.setValue("50");
        rightFtFilter.setCondition(SqlFilterCondition.EQUALS);
        FilterLogic extraFilterLogic = new FilterLogic(SqlFilterLogic.AND, leftFtFilter, rightFtFilter);
        Filter extraFilter = new Filter();
        extraFilter.setFeature(new Feature("ft_name"));
        extraFilter.setValue("100");
        extraFilter.setCondition(SqlFilterCondition.GREATER_THAN);

        // create training data
        fv.createTrainTestSplit(null, null, trainStart, trainEnd, testStart,
        testEnd,  description, DataFormat.CSV, coalesce, storageConnector, location, seed, statisticsConfig,
        writeOptions, extraFilterLogic, extraFilter);

        // or based on random split
        fv.createTrainTestSplit(20, 10, null, null,  null, null, description, DataFormat.CSV, coalesce,
        storageConnector, location, seed, statisticsConfig, writeOptions, extraFilterLogic, extraFilter);

 
 

Parameters:

testSize - Size of test set.

trainStart - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

trainEnd - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

testStart - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

testEnd - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

description - A string describing the contents of the training dataset to improve discoverability for Data Scientists.

dataFormat - The data format used to save the training dataset.

coalesce - If true the training dataset data will be coalesced into a single partition before writing. The resulting training dataset will be a single file per split.

storageConnector - Storage connector defining the sink location for the training dataset. If `null` is provided and materializes training dataset on HopsFS.

location - Path to complement the sink storage connector with, e.g if the storage connector points to an S3 bucket, this path can be used to define a sub-directory inside the bucket to place the training dataset. If empty string is provided `""`, saving the training dataset at the root defined by the storage connector.

seed - Define a seed to create the random splits with, in order to guarantee reproducability,

statisticsConfig - A configuration object, to generally enable descriptive statistics computation for this feature group, `"correlations`" to turn on feature correlation computation, `"histograms"` to compute feature value frequencies and `"exact_uniqueness"` to compute uniqueness, distinctness and entropy. The values should be booleans indicating the setting. To fully turn off statistics computation pass `statisticsConfig=null`.

writeOptions - Additional write options as key-value pairs.

extraFilterLogic - Additional filters (set of Filter objects) to be attached to the training dataset. The filters will be also applied in `getBatchData`.

extraFilter - Additional filter to be attached to the training dataset. The filter will be also applied in `getBatchData`.

Returns:

Integer Training dataset version.

Throws:

FeatureStoreException - If Client is not connected to Hopsworks and/or unable to identify format of the provided date strings to date formats.

IOException - Generic IO exception.

ParseException - In case it's unable to parse provided date strings to date types.

createTrainValidationTestSplit

public Integer createTrainValidationTestSplit(Float validationSize,
                                              Float testSize,
                                              String trainStart,
                                              String trainEnd,
                                              String validationStart,
                                              String validationEnd,
                                              String testStart,
                                              String testEnd,
                                              String description,
                                              DataFormat dataFormat)
                                       throws IOException,
                                              FeatureStoreException,
                                              ParseException

Create the metadata for a training dataset and save the corresponding training data into `location`. The training data is split into train, validation, and test set at random or according to time range. The training data can be retrieved by calling `featureView.getTrainValidationTestSplit`.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // create training dataset based on time split
        String trainStart = "20220101000000";
        String trainEnd = "20220630235959";
        String validationStart = "20220701000000";
        String validationEnd = "20220830235959";
        String testStart = "20220901000000";
        String testEnd = "20220931235959";
        String description = "demo training dataset":
        fv.createTrainTestSplit(null, null, trainStart, trainEnd, validationStart, validationEnd, testStart,
        testEnd, description, DataFormat.CSV);

        // or based on random split
        fv.createTrainTestSplit(20, 10, null, null, null, null, null, null, description, DataFormat.CSV);
 
 

Parameters:

validationSize - Size of validation set.

testSize - Size of test set.

trainStart - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

trainEnd - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

validationStart - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

validationEnd - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

testStart - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

testEnd - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

description - A string describing the contents of the training dataset to improve discoverability for Data Scientists.

dataFormat - The data format used to save the training dataset.

Returns:

Integer Training dataset version.

Throws:

FeatureStoreException - If Client is not connected to Hopsworks and/or unable to identify format of the provided date strings to date formats.

IOException - Generic IO exception.

ParseException - In case it's unable to parse provided date strings to date types.

createTrainValidationTestSplit

public Integer createTrainValidationTestSplit(Float validationSize,
                                              Float testSize,
                                              String trainStart,
                                              String trainEnd,
                                              String validationStart,
                                              String validationEnd,
                                              String testStart,
                                              String testEnd,
                                              String description,
                                              DataFormat dataFormat,
                                              Boolean coalesce,
                                              StorageConnector storageConnector,
                                              String location,
                                              Long seed,
                                              StatisticsConfig statisticsConfig,
                                              Map<String,String> writeOptions,
                                              FilterLogic extraFilterLogic,
                                              Filter extraFilter)
                                       throws IOException,
                                              FeatureStoreException,
                                              ParseException

Create the metadata for a training dataset and save the corresponding training data into `location`. The training data is split into train, validation, and test set at random or according to time range. The training data can be retrieved by calling `feature_view.getTrainValidationTestSplit`.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // create training dataset based on time split
        String trainStart = "20220101000000";
        String trainEnd = "20220630235959";
        String validationStart = "20220701000000";
        String validationEnd = "20220830235959";
        String testStart = "20220901000000";
        String testEnd = "20220931235959";
        String description = "demo training dataset":
        StorageConnector.S3Connector storageConnector = fs.getS3Connector("s3Connector");
        String location = "";
        Long seed = 1234L;
        Boolean coalesce = true;
        StatisticsConfig statisticsConfig = new StatisticsConfig(true, true, true, true)
        Map<String, String> writeOptions = new HashMap<String, String>() {{
                           put("header", "true");
                           put("delimiter", ",")}
                           };
        // define extra filters
        Filter leftFtFilter = new Filter();
        leftFtFilter.setFeature(new Feature("left_ft_name"));
        leftFtFilter.setValue("400");
        leftFtFilter.setCondition(SqlFilterCondition.EQUALS);
        Filter rightFtFilter = new Filter();
        rightFtFilter.setFeature(new Feature("right_ft_name"));
        rightFtFilter.setValue("50");
        rightFtFilter.setCondition(SqlFilterCondition.EQUALS);
        FilterLogic extraFilterLogic = new FilterLogic(SqlFilterLogic.AND, leftFtFilter, rightFtFilter);
        Filter extraFilter = new Filter();
        extraFilter.setFeature(new Feature("ft_name"));
        extraFilter.setValue("100");
        extraFilter.setCondition(SqlFilterCondition.GREATER_THAN);
        // create training data
        fv.createTrainTestSplit(null, null, trainStart, trainEnd, validationStart, validationEnd, testStart,
        testEnd,  description, DataFormat.CSV, coalesce, storageConnector, location, seed, statisticsConfig,
        writeOptions, extraFilterLogic, extraFilter);

        // or based on random split
        fv.createTrainTestSplit(20, 10, null, null, null, null, null, null, description, DataFormat.CSV, coalesce,
        storageConnector, location, seed, statisticsConfig, writeOptions, extraFilterLogic, extraFilter);
 
 

Parameters:

validationSize - Size of validation set.

testSize - Size of test set.

trainStart - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

trainEnd - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

validationStart - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

validationEnd - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

testStart - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

testEnd - Datetime string. The String should be formatted in one of the following formats `yyyyMMdd`, `yyyyMMddHH`, `yyyyMMddHHmm`, or `yyyyMMddHHmmss`.

description - A string describing the contents of the training dataset to improve discoverability for Data Scientists.

dataFormat - The data format used to save the training dataset.

coalesce - If true the training dataset data will be coalesced into a single partition before writing. The resulting training dataset will be a single file per split.

storageConnector - Storage connector defining the sink location for the training dataset. If `null` is provided and materializes training dataset on HopsFS.

location - Path to complement the sink storage connector with, e.g if the storage connector points to an S3 bucket, this path can be used to define a sub-directory inside the bucket to place the training dataset. If empty string is provided `""`, saving the training dataset at the root defined by the storage connector.

seed - Define a seed to create the random splits with, in order to guarantee reproducability,

statisticsConfig - A configuration object, to generally enable descriptive statistics computation for this feature group, `"correlations`" to turn on feature correlation computation, `"histograms"` to compute feature value frequencies and `"exact_uniqueness"` to compute uniqueness, distinctness and entropy. The values should be booleans indicating the setting. To fully turn off statistics computation pass `statisticsConfig=null`.

writeOptions - Additional write options as key-value pairs.

extraFilterLogic - Additional filters (set of Filter objects) to be attached to the training dataset. The filters will be also applied in `getBatchData`.

extraFilter - Additional filter to be attached to the training dataset. The filter will be also applied in `getBatchData`.

Returns:

Integer Training dataset version.

Throws:

FeatureStoreException - If Client is not connected to Hopsworks and/or unable to identify format of the provided date strings to date formats.

IOException - Generic IO exception.

ParseException - In case it's unable to parse provided date strings to date types.

validateTrainTestSplit

protected void validateTrainTestSplit(Float testSize,
                                      String trainEnd,
                                      String testStart)
                               throws FeatureStoreException

Throws:

FeatureStoreException

validateTrainValidationTestSplit

protected void validateTrainValidationTestSplit(Float validationSize,
                                                Float testSize,
                                                String trainEnd,
                                                String validationStart,
                                                String validationEnd,
                                                String testStart)
                                         throws FeatureStoreException

Throws:

FeatureStoreException

initServing

public void initServing()
                 throws FeatureStoreException,
                        IOException,
                        SQLException,
                        ClassNotFoundException

Initialise feature view to retrieve feature vector from online feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // Initialise feature view serving
        fv.initServing();
 
 

Throws:

FeatureStoreException - In case client is not connected to Hopsworks.

IOException - Generic IO exception.

SQLException - In case there is online storage (RonDB) access error or other errors.

ClassNotFoundException - In case class `com.mysql.jdbc.Driver` can not be found.

initServing

public void initServing(Boolean batch,
                        Boolean external)
                 throws FeatureStoreException,
                        IOException,
                        SQLException,
                        ClassNotFoundException

Initialise feature view to retrieve feature vector from online feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // Initialise feature view batch serving
        fv.initServing(true, false);
 
 

Parameters:

batch - Whether to initialise feature view to retrieve feature vector from offline feature store.

external - If set to `true`, the connection to the online feature store is established using the same host as for the `host` parameter in the connection object. If set to False, the online feature store storage connector is used which relies on the private IP.

Throws:

FeatureStoreException - In case client is not connected to Hopsworks.

IOException - Generic IO exception.

SQLException - In case there is online storage (RonDB) access error or other errors.

ClassNotFoundException - In case class `com.mysql.jdbc.Driver` can not be found.

initBatchScoring

public void initBatchScoring(Integer trainingDatasetVersion)

Initialise feature view to retrieve feature vector from offline feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // Initialise feature view batch scoring
        fv.initBatchScoring(1);
 
 

Parameters:

trainingDatasetVersion - Version of training dataset to identify additional filters attached to the training dataset and statistics to use for transformation functions.

getFeatureVector

public List<Object> getFeatureVector(Map<String,Object> entry)
                              throws FeatureStoreException,
                                     IOException,
                                     ClassNotFoundException

Returns assembled feature vector from online feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // define primary key values to fetch data from online feature store
        Map<String, Object> pkMap = new HashMap<String, Object>() {
               {put("customer_id", 1);
                put("contract_id" , 100);
                }
        };
        // get feature vector
        fv.getFeatureVector(entry);
 
 

Parameters:

entry - Fictionary of feature group primary key and values provided by serving application.

Returns:

List of feature values related to provided primary keys, ordered according to positions of the features in the feature view query.

Throws:

FeatureStoreException - In case client is not connected to Hopsworks.

IOException - Generic IO exception.

ClassNotFoundException - In case class `com.mysql.jdbc.Driver` can not be found.

getFeatureVector

public List<Object> getFeatureVector(Map<String,Object> entry,
                                     boolean external)
                              throws FeatureStoreException,
                                     IOException,
                                     ClassNotFoundException

Returns assembled feature vector from online feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // define primary key values to fetch data from online feature store
        Map<String, Object> pkMap = new HashMap<String, Object>() {
               {put("customer_id", 1);
                put("contract_id" , 100);
                }
        };
        // get feature vector
        fv.getFeatureVector(entry, false);
 
 

Parameters:

entry - Dictionary of feature group primary key and values provided by serving application.

external - If set to true, the connection to the online feature store is established using the same host as for the `host` parameter in the connection object. If set to false, the online feature store storage connector is used which relies on the private IP. Defaults to True if connection to Hopsworks is established from external environment

Returns:

List of feature values related to provided primary keys, ordered according to positions of the features in the feature view query.

Throws:

FeatureStoreException - In case client is not connected to Hopsworks.

IOException - Generic IO exception.

ClassNotFoundException - In case class `com.mysql.jdbc.Driver` can not be found.

getFeatureVectors

public List<List<Object>> getFeatureVectors(Map<String,List<Object>> entry)
                                     throws SQLException,
                                            FeatureStoreException,
                                            IOException

Returns assembled feature vectors in batches from online feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // define primary key values to fetch data from online feature store
        Map<String, List<Long>> entry = ...;
        // get feature vector
        fv.getFeatureVector(entry);
 
 

Parameters:

entry - A list of dictionaries of feature group primary key and values provided by serving application.

Returns:

List of lists of feature values related to provided primary keys, ordered according to positions of the features in the feature view query.

Throws:

FeatureStoreException - In case client is not connected to Hopsworks.

IOException - Generic IO exception.

SQLException - In case there is online storage (RonDB) access error or other errors.

getFeatureVectors

public List<List<Object>> getFeatureVectors(Map<String,List<Object>> entry,
                                            boolean external)
                                     throws SQLException,
                                            FeatureStoreException,
                                            IOException,
                                            ClassNotFoundException

Returns assembled feature vectors in batches from online feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // define primary key values to fetch data from online feature store
        Map<String, List<Long>> entry = ...;
        // get feature vector
        fv.getFeatureVectors(entry, false);
 
 

Parameters:

entry - A list of dictionaries of feature group primary key and values provided by serving application.

external - If set to `true`, the connection to the online feature store is established using the same host as for the `host` parameter in the connection object. If set to False, the online feature store storage connector is used which relies on the private IP.

Returns:

List of lists of feature values related to provided primary keys, ordered according to positions of this features in the feature view query.

Throws:

FeatureStoreException - In case client is not connected to Hopsworks.

IOException - Generic IO exception.

SQLException - In case there is online storage (RonDB) access error or other errors.

ClassNotFoundException - In case class `com.mysql.jdbc.Driver` can not be found.

addTag

public void addTag(String name,
                   Object value)
            throws FeatureStoreException,
                   IOException

Add name/value tag to the feature view. A tag consists of a name and value pair. Tag names are unique identifiers across the whole cluster. The value of a tag can be any valid json - primitives, arrays or json objects.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // attach a tag to a feature view
        JSONObject value = ...;
        fv.addTag("tag_schema", value);
 
 

Parameters:

name - Name of the tag

value - Value of the tag. The value of a tag can be any valid json - primitives, arrays or json objects

Throws:

FeatureStoreException - If Client is not connected to Hopsworks.

IOException - Generic IO exception.

getTags

public Map<String,Object> getTags()
                           throws FeatureStoreException,
                                  IOException

Get all tags of the feature view.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // get tags
        fv.getTags();
 
 

Returns:

Map<String, Object> a map of tag name and values. The value of a tag can be any valid json - primitives, arrays or json objects

Throws:

FeatureStoreException - If Client is not connected to Hopsworks.

IOException - Generic IO exception.

getTag

public Object getTag(String name)
              throws FeatureStoreException,
                     IOException

Get a single tag value of the feature view.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // get tag
        fv.getTag("tag_name");
 
 

Parameters:

name - name of the tag

Returns:

Object The value of a tag can be any valid json - primitives, arrays or json objects

Throws:

FeatureStoreException - If Client is not connected to Hopsworks.

IOException - Generic IO exception.

deleteTag

public void deleteTag(String name)
               throws FeatureStoreException,
                      IOException

Delete a tag of the feature view.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // delete tag
        fv.deleteTag("tag_name");
 
 

Parameters:

name - Name of the tag to be deleted.

Throws:

FeatureStoreException - If Client is not connected to Hopsworks.

IOException - Generic IO exception.

addTrainingDatasetTag

public void addTrainingDatasetTag(Integer version,
                                  String name,
                                  Object value)
                           throws FeatureStoreException,
                                  IOException

Add name/value tag to the training dataset.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // add tag to datasets version 1 in this feature view.
        JSONObject json = ...;
        fv.addTrainingDatasetTag(1, "tag_name", json);
 
 

Parameters:

version - Training dataset version.

name - Name of the tag.

value - Value of the tag. The value of a tag can be any valid json - primitives, arrays or json objects.

Throws:

FeatureStoreException - If Client is not connected to Hopsworks.

IOException - Generic IO exception.

getTrainingDatasetTags

public Map<String,Object> getTrainingDatasetTags(Integer version)
                                          throws FeatureStoreException,
                                                 IOException

Get all tags of the training dataset.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // get tags of training dataset version 1 in this feature view.
        fv.getTrainingDatasetTags(1);
 
 

Parameters:

version - Training dataset version.

Returns:

Map<String, Object> A map of tag name and values. The value of a tag can be any valid json - primitives, arrays or json objects

Throws:

FeatureStoreException - If Client is not connected to Hopsworks.

IOException - Generic IO exception.

getTrainingDatasetTag

public Object getTrainingDatasetTag(Integer version,
                                    String name)
                             throws FeatureStoreException,
                                    IOException

Get a single tag value of the training dataset.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // get tag with name `"demo_name"` of training dataset version 1 in this feature view.
        fv.getTrainingDatasetTags(1, "demo_name");
 
 

Parameters:

version - Training dataset version.

name - Name of the tag.

Returns:

Object The value of a tag can be any valid json - primitives, arrays or json objects.

Throws:

FeatureStoreException - If Client is not connected to Hopsworks.

IOException - Generic IO exception.

deleteTrainingDatasetTag

public void deleteTrainingDatasetTag(Integer version,
                                     String name)
                              throws FeatureStoreException,
                                     IOException

Delete a tag of the training dataset.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // delete tag with name `"demo_name"` of training dataset version 1 in this feature view.
        fv.deleteTrainingDatasetTag(1, "demo_name");
 
 

Parameters:

version - Tag version.

name - Name of the tag to be deleted.

Throws:

FeatureStoreException - If Client is not connected to Hopsworks.

IOException - Generic IO exception.

delete

public void delete()
            throws FeatureStoreException,
                   IOException

Delete current feature view, all associated metadata and training data.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature view handle
        FeatureView fv = fs.getFeatureView("fv_name", 1);
        // delete feature view
        fv.delete();
 
 

Throws:

FeatureStoreException - In case client is not connected to Hopsworks.

IOException - Generic IO exception.

getPrimaryKeys

public HashSet<String> getPrimaryKeys()
                               throws SQLException,
                                      IOException,
                                      FeatureStoreException,
                                      ClassNotFoundException

Set of primary key names that is used as keys in input dict object for `get_serving_vector` method.

Returns:

Set of serving keys

Throws:

SQLException

IOException

FeatureStoreException

ClassNotFoundException

closeVectorServer

public void closeVectorServer()

Closes the ExecutorService and JDBC DataSource used to retrieve feature vectors from the online feature store.