com.logicalclocks.hsfs.spark

Class TrainingDataset

java.lang.Object

com.logicalclocks.hsfs.TrainingDatasetBase

com.logicalclocks.hsfs.spark.TrainingDataset

public class TrainingDataset
extends TrainingDatasetBase

Field Summary

Fields inherited from class com.logicalclocks.hsfs.TrainingDatasetBase

coalesce, dataFormat, description, eventEndTime, eventStartTime, extraFilter, features, featureStore, id, label, location, LOGGER, name, seed, splits, statisticsConfig, storageConnector, trainingDatasetType, trainSplit, type, vectorServer, version

Constructor Summary

Constructors

Constructor and Description

TrainingDataset(@NonNull String name, Integer version, String description, DataFormat dataFormat, Boolean coalesce, StorageConnector storageConnector, String location, List<Split> splits, String trainSplit, Long seed, FeatureStore featureStore, StatisticsConfig statisticsConfig, List<String> label, String eventStartTime, String eventEndTime, TrainingDatasetType trainingDatasetType, Float validationSize, Float testSize, String trainStart, String trainEnd, String validationStart, String validationEnd, String testStart, String testEnd, Integer timeSplitSize, FilterLogic extraFilterLogic, Filter extraFilter)

Method Summary

Modifier and Type	Method and Description
void	addTag(String name, Object value) Add name/value tag to the training dataset.
Statistics	computeStatistics() Recompute the statistics for the entire training dataset and save them to the feature store.
void	delete() Delete training dataset and all associated metadata.
void	deleteTag(String name) Delete a tag of the training dataset.
String	getQuery(Storage storage, boolean withLabel)
List<Object>	getServingVector(Map<String,Object> entry) Retrieve feature vector from online feature store.
Statistics	getStatistics() Get the last statistics commit for the training dataset.
Statistics	getStatistics(String commitTime) Get the statistics of a specific commit time for the training dataset.
Object	getTag(String name) Get a single tag value of the training dataset.
Map<String,Object>	getTags() Get all tags of the training dataset.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read() Read the content of the training dataset.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(Map<String,String> readOptions) Read the content of the training dataset.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(String split) Read all a single split from the training dataset.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(String split, Map<String,String> readOptions) Read a single split from the training dataset.
void	save(Query query)
void	save(Query query, Map<String,String> writeOptions) Create the training dataset based on the content of the feature store query.
void	show(int numRows) Show numRows from the training dataset (across all splits).
void	updateStatisticsConfig() Update the statistics configuration of the training dataset.

Methods inherited from class com.logicalclocks.hsfs.TrainingDatasetBase

getLabel, getQuery, getQuery, getQuery, getServingKeys, getServingVector, getServingVectors, getServingVectors, getTrainingDatasetType, initPreparedStatement, initPreparedStatement, initPreparedStatement, setLabel, setTimeSeriesSplits, setValTestSplit

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TrainingDataset

public TrainingDataset(@NonNull
                       @NonNull String name,
                       Integer version,
                       String description,
                       DataFormat dataFormat,
                       Boolean coalesce,
                       StorageConnector storageConnector,
                       String location,
                       List<Split> splits,
                       String trainSplit,
                       Long seed,
                       FeatureStore featureStore,
                       StatisticsConfig statisticsConfig,
                       List<String> label,
                       String eventStartTime,
                       String eventEndTime,
                       TrainingDatasetType trainingDatasetType,
                       Float validationSize,
                       Float testSize,
                       String trainStart,
                       String trainEnd,
                       String validationStart,
                       String validationEnd,
                       String testStart,
                       String testEnd,
                       Integer timeSplitSize,
                       FilterLogic extraFilterLogic,
                       Filter extraFilter)
                throws FeatureStoreException,
                       ParseException

Throws:

FeatureStoreException

ParseException

Method Detail

save

public void save(Query query)
          throws FeatureStoreException,
                 IOException

Throws:

FeatureStoreException

IOException

save

public void save(Query query,
                 Map<String,String> writeOptions)
          throws FeatureStoreException,
                 IOException

Create the training dataset based on the content of the feature store query.

Parameters:

query - the query to save as training dataset

writeOptions - options to pass to the Spark write operation

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read()
                                                            throws FeatureStoreException,
                                                                   IOException

Read the content of the training dataset.

Specified by:

read in class TrainingDatasetBase

Returns:

Spark Dataset containing the training dataset data

Throws:

FeatureStoreException - if the training dataset has splits and the split was not specified

IOException - IOException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(Map<String,String> readOptions)
                                                            throws FeatureStoreException,
                                                                   IOException

Read the content of the training dataset.

Parameters:

readOptions - options to pass to the Spark read operation

Returns:

Spark Dataset containing the training dataset data

Throws:

FeatureStoreException - if the training dataset has splits and the split was not specified

IOException - IOException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(String split)
                                                            throws FeatureStoreException,
                                                                   IOException

Read all a single split from the training dataset.

Parameters:

split - the split name

Returns:

Spark Dataset containing the training dataset data

Throws:

FeatureStoreException - if the training dataset has splits and the split was not specified

IOException - IOException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(String split,
                                                                   Map<String,String> readOptions)
                                                            throws FeatureStoreException,
                                                                   IOException

Read a single split from the training dataset.

Specified by:

read in class TrainingDatasetBase

Parameters:

split - the split name

readOptions - options to pass to the Spark read operation

Returns:

Spark Dataset containing the training dataset data

Throws:

FeatureStoreException - if the training dataset has splits and the split was not specified

IOException - IOException

show

public void show(int numRows)
          throws FeatureStoreException,
                 IOException

Show numRows from the training dataset (across all splits).

Specified by:

show in class TrainingDatasetBase

Parameters:

numRows - number of rows to display

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

computeStatistics

public Statistics computeStatistics()
                             throws FeatureStoreException,
                                    IOException

Recompute the statistics for the entire training dataset and save them to the feature store.

Specified by:

computeStatistics in class TrainingDatasetBase

Returns:

statistics object of computed statistics

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

updateStatisticsConfig

public void updateStatisticsConfig()
                            throws FeatureStoreException,
                                   IOException

Update the statistics configuration of the training dataset. Change the `enabled`, `histograms`, `correlations` or `columns` attributes and persist the changes by calling this method.

Specified by:

updateStatisticsConfig in class TrainingDatasetBase

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

getStatistics

public Statistics getStatistics()
                         throws FeatureStoreException,
                                IOException

Get the last statistics commit for the training dataset.

Specified by:

getStatistics in class TrainingDatasetBase

Returns:

statistics object of latest commit

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

getStatistics

public Statistics getStatistics(String commitTime)
                         throws FeatureStoreException,
                                IOException

Get the statistics of a specific commit time for the training dataset.

Specified by:

getStatistics in class TrainingDatasetBase

Parameters:

commitTime - commit time in the format "YYYYMMDDhhmmss"

Returns:

statistics object for the commit time

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

addTag

public void addTag(String name,
                   Object value)
            throws FeatureStoreException,
                   IOException

Add name/value tag to the training dataset.

Specified by:

addTag in class TrainingDatasetBase

Parameters:

name - name of the tag

value - value of the tag. The value of a tag can be any valid json - primitives, arrays or json objects

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

getTags

public Map<String,Object> getTags()
                           throws FeatureStoreException,
                                  IOException

Get all tags of the training dataset.

Specified by:

getTags in class TrainingDatasetBase

Returns:

a map of tag name and values. The value of a tag can be any valid json - primitives, arrays or json objects

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

getTag

public Object getTag(String name)
              throws FeatureStoreException,
                     IOException

Get a single tag value of the training dataset.

Specified by:

getTag in class TrainingDatasetBase

Parameters:

name - name of the tag

Returns:

The value of a tag can be any valid json - primitives, arrays or json objects

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

deleteTag

public void deleteTag(String name)
               throws FeatureStoreException,
                      IOException

Delete a tag of the training dataset.

Specified by:

deleteTag in class TrainingDatasetBase

Parameters:

name - name of the tag to be deleted

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

getQuery

public String getQuery(Storage storage,
                       boolean withLabel)
                throws FeatureStoreException,
                       IOException

Specified by:

getQuery in class TrainingDatasetBase

Throws:

FeatureStoreException

IOException

getServingVector

public List<Object> getServingVector(Map<String,Object> entry)
                              throws SQLException,
                                     FeatureStoreException,
                                     IOException,
                                     ClassNotFoundException

Retrieve feature vector from online feature store.

Overrides:

getServingVector in class TrainingDatasetBase

Parameters:

entry - Map object with kes as primary key names of the training dataset features groups and values as corresponding ids to retrieve feature vector from online feature store.

Returns:

list of feature values sorted according to provided primary keys.

Throws:

SQLException - SQLException

IOException - IOException

FeatureStoreException - FeatureStoreException

ClassNotFoundException - ClassNotFoundException

delete

public void delete()
            throws FeatureStoreException,
                   IOException

Delete training dataset and all associated metadata. Note that this operation drops only files which were materialized in HopsFS. If you used a Storage Connector for a cloud storage such as S3, the data will not be deleted, but you will not be able to track it anymore from the Feature Store. This operation drops all metadata associated with this version of the training dataset and and the materialized data in HopsFS.

Specified by:

delete in class TrainingDatasetBase

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException