com.logicalclocks.hsfs

Class TrainingDatasetBase

java.lang.Object

com.logicalclocks.hsfs.TrainingDatasetBase

Direct Known Subclasses:

TrainingDataset

public abstract class TrainingDatasetBase
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
protected Boolean	coalesce
protected DataFormat	dataFormat
protected String	description
protected Date	eventEndTime
protected Date	eventStartTime
protected FilterLogic	extraFilter
protected List<TrainingDatasetFeature>	features
protected FeatureStoreBase	featureStore
protected Integer	id
protected List<String>	label
protected String	location
protected static org.slf4j.Logger	LOGGER
protected String	name
protected Long	seed
protected List<Split>	splits
protected StatisticsConfig	statisticsConfig
protected StorageConnector	storageConnector
protected TrainingDatasetType	trainingDatasetType
protected String	trainSplit
protected String	type
protected VectorServer	vectorServer
protected Integer	version

Constructor Summary

Constructors

Constructor and Description
TrainingDatasetBase()

Method Summary

Modifier and Type	Method and Description
abstract void	addTag(String name, Object value) Add name/value tag to the training dataset.
abstract Statistics	computeStatistics() Recompute the statistics for the entire training dataset and save them to the feature store.
abstract void	delete() Delete training dataset and all associated metadata.
abstract void	deleteTag(String name) Delete a tag of the training dataset.
List<String>	getLabel()
String	getQuery()
String	getQuery(boolean withLabel)
String	getQuery(Storage storage)
abstract String	getQuery(Storage storage, boolean withLabel)
HashSet<String>	getServingKeys() Set of primary key names that is used as keys in input dict object for `get_serving_vector` method.
List<Object>	getServingVector(Map<String,Object> entry) Retrieve feature vector from online feature store.
List<Object>	getServingVector(Map<String,Object> entry, boolean external) Retrieve feature vector from online feature store.
List<List<Object>>	getServingVectors(Map<String,List<Object>> entry)
List<List<Object>>	getServingVectors(Map<String,List<Object>> entry, boolean external)
abstract Statistics	getStatistics() Get the last statistics commit for the training dataset.
abstract Statistics	getStatistics(String commitTime) Get the statistics of a specific commit time for the training dataset.
abstract Object	getTag(String name) Get a single tag value of the training dataset.
abstract Map<String,Object>	getTags() Get all tags of the training dataset.
TrainingDatasetType	getTrainingDatasetType(StorageConnector storageConnector)
void	initPreparedStatement() Initialise and cache parametrised prepared statement to retrieve feature vector from online feature store.
void	initPreparedStatement(boolean external) Initialise and cache parametrised prepared statement to retrieve feature vector from online feature store.
void	initPreparedStatement(boolean external, boolean batch) Initialise and cache parametrised prepared statement to retrieve batch feature vectors from online feature store.
abstract <T> T	read() Read the content of the training dataset.
abstract Object	read(String split, Map<String,String> readOptions) Read a single split from the training dataset.
void	setLabel(List<String> label)
void	setTimeSeriesSplits(Integer timeSplitSize, String trainStart, String trainEnd, String valStart, String valEnd, String testStart, String testEnd)
void	setValTestSplit(Float valSize, Float testSize)
abstract void	show(int numRows) Show numRows from the training dataset (across all splits).
abstract void	updateStatisticsConfig() Update the statistics configuration of the training dataset.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

id

protected Integer id

name

protected String name

version

protected Integer version

description

protected String description

coalesce

protected Boolean coalesce

trainingDatasetType

protected TrainingDatasetType trainingDatasetType

features

protected List<TrainingDatasetFeature> features

featureStore

protected FeatureStoreBase featureStore

location

protected String location

seed

protected Long seed

splits

protected List<Split> splits

trainSplit

protected String trainSplit

label

protected List<String> label

eventStartTime

protected Date eventStartTime

eventEndTime

protected Date eventEndTime

extraFilter

protected FilterLogic extraFilter

dataFormat

protected DataFormat dataFormat

storageConnector

protected StorageConnector storageConnector

statisticsConfig

protected StatisticsConfig statisticsConfig

type

protected String type

LOGGER

protected static final org.slf4j.Logger LOGGER

vectorServer

protected VectorServer vectorServer

Constructor Detail

TrainingDatasetBase

public TrainingDatasetBase()

Method Detail

setTimeSeriesSplits

public void setTimeSeriesSplits(Integer timeSplitSize,
                                String trainStart,
                                String trainEnd,
                                String valStart,
                                String valEnd,
                                String testStart,
                                String testEnd)
                         throws FeatureStoreException,
                                ParseException

Throws:

FeatureStoreException

ParseException

setValTestSplit

public void setValTestSplit(Float valSize,
                            Float testSize)

getQuery

public String getQuery()
                throws FeatureStoreException,
                       IOException

Throws:

FeatureStoreException

IOException

getQuery

public String getQuery(boolean withLabel)
                throws FeatureStoreException,
                       IOException

Throws:

FeatureStoreException

IOException

getQuery

public String getQuery(Storage storage)
                throws FeatureStoreException,
                       IOException

Throws:

FeatureStoreException

IOException

getQuery

public abstract String getQuery(Storage storage,
                                boolean withLabel)
                         throws FeatureStoreException,
                                IOException

Throws:

FeatureStoreException

IOException

getLabel

public List<String> getLabel()

setLabel

public void setLabel(List<String> label)

initPreparedStatement

public void initPreparedStatement()
                           throws SQLException,
                                  IOException,
                                  FeatureStoreException,
                                  ClassNotFoundException

Initialise and cache parametrised prepared statement to retrieve feature vector from online feature store.

Throws:

SQLException

IOException

FeatureStoreException

ClassNotFoundException

initPreparedStatement

public void initPreparedStatement(boolean external)
                           throws SQLException,
                                  IOException,
                                  FeatureStoreException,
                                  ClassNotFoundException

Initialise and cache parametrised prepared statement to retrieve feature vector from online feature store.

Throws:

SQLException

IOException

FeatureStoreException

ClassNotFoundException

initPreparedStatement

public void initPreparedStatement(boolean external,
                                  boolean batch)
                           throws SQLException,
                                  IOException,
                                  FeatureStoreException,
                                  ClassNotFoundException

Initialise and cache parametrised prepared statement to retrieve batch feature vectors from online feature store.

Throws:

SQLException

IOException

FeatureStoreException

ClassNotFoundException

getServingVector

public List<Object> getServingVector(Map<String,Object> entry)
                              throws SQLException,
                                     FeatureStoreException,
                                     IOException,
                                     ClassNotFoundException

Retrieve feature vector from online feature store.

Parameters:

entry - Map object with kes as primary key names of the training dataset features groups and values as corresponding ids to retrieve feature vector from online feature store.

Throws:

FeatureStoreException

IOException

SQLException

ClassNotFoundException

getServingVector

public List<Object> getServingVector(Map<String,Object> entry,
                                     boolean external)
                              throws SQLException,
                                     FeatureStoreException,
                                     IOException,
                                     ClassNotFoundException

Retrieve feature vector from online feature store.

Parameters:

entry - Map object with kes as primary key names of the training dataset features groups and values as corresponding ids to retrieve feature vector from online feature store.

external - If true, the connection to the online feature store will be established using the hostname provided in the hsfs.connection() setup.

Throws:

FeatureStoreException

IOException

SQLException

ClassNotFoundException

getServingVectors

public List<List<Object>> getServingVectors(Map<String,List<Object>> entry)
                                     throws SQLException,
                                            FeatureStoreException,
                                            IOException,
                                            ClassNotFoundException

Throws:

SQLException

FeatureStoreException

IOException

ClassNotFoundException

getServingVectors

public List<List<Object>> getServingVectors(Map<String,List<Object>> entry,
                                            boolean external)
                                     throws SQLException,
                                            FeatureStoreException,
                                            IOException,
                                            ClassNotFoundException

Throws:

SQLException

FeatureStoreException

IOException

ClassNotFoundException

getServingKeys

public HashSet<String> getServingKeys()
                               throws SQLException,
                                      IOException,
                                      FeatureStoreException,
                                      ClassNotFoundException

Set of primary key names that is used as keys in input dict object for `get_serving_vector` method.

Returns:

Set of serving keys

Throws:

SQLException

IOException

FeatureStoreException

ClassNotFoundException

getTrainingDatasetType

public TrainingDatasetType getTrainingDatasetType(StorageConnector storageConnector)

read

public abstract <T> T read()
                    throws FeatureStoreException,
                           IOException

Read the content of the training dataset.

Returns:

Spark Dataset containing the training dataset data

Throws:

FeatureStoreException - if the training dataset has splits and the split was not specified

IOException

read

public abstract Object read(String split,
                            Map<String,String> readOptions)
                     throws FeatureStoreException,
                            IOException

Read a single split from the training dataset.

Parameters:

split - the split name

readOptions - options to pass to the Spark read operation

Returns:

Spark Dataset containing the training dataset data

Throws:

FeatureStoreException - if the training dataset has splits and the split was not specified

IOException

show

public abstract void show(int numRows)
                   throws FeatureStoreException,
                          IOException

Show numRows from the training dataset (across all splits).

Parameters:

numRows -

Throws:

FeatureStoreException

IOException

computeStatistics

public abstract Statistics computeStatistics()
                                      throws FeatureStoreException,
                                             IOException

Recompute the statistics for the entire training dataset and save them to the feature store.

Returns:

statistics object of computed statistics

Throws:

FeatureStoreException

IOException

updateStatisticsConfig

public abstract void updateStatisticsConfig()
                                     throws FeatureStoreException,
                                            IOException

Update the statistics configuration of the training dataset. Change the `enabled`, `histograms`, `correlations` or `columns` attributes and persist the changes by calling this method.

Throws:

FeatureStoreException

IOException

getStatistics

public abstract Statistics getStatistics()
                                  throws FeatureStoreException,
                                         IOException

Get the last statistics commit for the training dataset.

Returns:

statistics object of latest commit

Throws:

FeatureStoreException

IOException

getStatistics

public abstract Statistics getStatistics(String commitTime)
                                  throws FeatureStoreException,
                                         IOException

Get the statistics of a specific commit time for the training dataset.

Parameters:

commitTime - commit time in the format "YYYYMMDDhhmmss"

Returns:

statistics object for the commit time

Throws:

FeatureStoreException

IOException

addTag

public abstract void addTag(String name,
                            Object value)
                     throws FeatureStoreException,
                            IOException

Add name/value tag to the training dataset.

Parameters:

name - name of the tag

value - value of the tag. The value of a tag can be any valid json - primitives, arrays or json objects

Throws:

FeatureStoreException

IOException

getTags

public abstract Map<String,Object> getTags()
                                    throws FeatureStoreException,
                                           IOException

Get all tags of the training dataset.

Returns:

a map of tag name and values. The value of a tag can be any valid json - primitives, arrays or json objects

Throws:

FeatureStoreException

IOException

getTag

public abstract Object getTag(String name)
                       throws FeatureStoreException,
                              IOException

Get a single tag value of the training dataset.

Parameters:

name - name of the tag

Returns:

The value of a tag can be any valid json - primitives, arrays or json objects

Throws:

FeatureStoreException

IOException

deleteTag

public abstract void deleteTag(String name)
                        throws FeatureStoreException,
                               IOException

Delete a tag of the training dataset.

Parameters:

name - name of the tag to be deleted

Throws:

FeatureStoreException

IOException

delete

public abstract void delete()
                     throws FeatureStoreException,
                            IOException

Delete training dataset and all associated metadata. Note that this operation drops only files which were materialized in HopsFS. If you used a Storage Connector for a cloud storage such as S3, the data will not be deleted, but you will not be able to track it anymore from the Feature Store. This operation drops all metadata associated with this version of the training dataset and the materialized data in HopsFS.

Throws:

FeatureStoreException

IOException