Package com.logicalclocks.hsfs.spark

Class TrainingDataset

java.lang.Object

com.logicalclocks.hsfs.TrainingDatasetBase

com.logicalclocks.hsfs.spark.TrainingDataset

public class TrainingDataset
extends TrainingDatasetBase

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	TrainingDataset.TrainingDatasetBuilder

Field Summary

Fields inherited from class com.logicalclocks.hsfs.TrainingDatasetBase

coalesce, dataFormat, dataSource, description, eventEndTime, eventStartTime, extraFilter, features, featureStore, id, label, location, LOGGER, name, seed, splits, statisticsConfig, trainingDatasetType, trainSplit, type, version

Constructor Summary

Constructors

Constructor

Description

TrainingDataset(Integer version, String description, DataFormat dataFormat, Boolean coalesce, StorageConnector storageConnector, String location, List<Split> splits, String trainSplit, Long seed, FeatureStoreBase featureStore, StatisticsConfig statisticsConfig, List<String> label, String eventStartTime, String eventEndTime, TrainingDatasetType trainingDatasetType, Float validationSize, Float testSize, String trainStart, String trainEnd, String validationStart, String validationEnd, String testStart, String testEnd, Integer timeSplitSize, FilterLogic extraFilterLogic, Filter extraFilter, DataSource dataSource)

Method Summary

Modifier and Type	Method	Description
Statistics	computeStatistics()	Recompute the statistics for the entire training dataset and save them to the feature store.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read()	Read the content of the training dataset.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(String split)	Read all a single split from the training dataset.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(String split, Map<String,String> readOptions)	Read a single split from the training dataset.
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(Map<String,String> readOptions)	Read the content of the training dataset.
void	show(int numRows)	Show numRows from the training dataset (across all splits).

Methods inherited from class com.logicalclocks.hsfs.TrainingDatasetBase

addTag, delete, deleteTag, getLabel, getStatistics, getStatistics, getTag, getTags, getTrainingDatasetType, setLabel, setTimeSeriesSplits, setValTestSplit

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

TrainingDataset

public TrainingDataset(Integer version,
 String description,
 DataFormat dataFormat,
 Boolean coalesce,
 StorageConnector storageConnector,
 String location,
 List<Split> splits,
 String trainSplit,
 Long seed,
 FeatureStoreBase featureStore,
 StatisticsConfig statisticsConfig,
 List<String> label,
 String eventStartTime,
 String eventEndTime,
 TrainingDatasetType trainingDatasetType,
 Float validationSize,
 Float testSize,
 String trainStart,
 String trainEnd,
 String validationStart,
 String validationEnd,
 String testStart,
 String testEnd,
 Integer timeSplitSize,
 FilterLogic extraFilterLogic,
 Filter extraFilter,
 DataSource dataSource)
                throws FeatureStoreException,
ParseException

Throws:

FeatureStoreException

ParseException

Method Details

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read()
                                                            throws FeatureStoreException,
IOException

Read the content of the training dataset.

Returns:

Spark Dataset containing the training dataset data

Throws:

FeatureStoreException - if the training dataset has splits and the split was not specified

IOException - IOException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(Map<String,String> readOptions)
                                                            throws FeatureStoreException,
IOException

Read the content of the training dataset.

Parameters:

readOptions - options to pass to the Spark read operation

Returns:

Spark Dataset containing the training dataset data

Throws:

FeatureStoreException - if the training dataset has splits and the split was not specified

IOException - IOException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(String split)
                                                            throws FeatureStoreException,
IOException

Read all a single split from the training dataset.

Parameters:

split - the split name

Returns:

Spark Dataset containing the training dataset data

Throws:

FeatureStoreException - if the training dataset has splits and the split was not specified

IOException - IOException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(String split,
 Map<String,String> readOptions)
                                                            throws FeatureStoreException,
IOException

Read a single split from the training dataset.

Parameters:

split - the split name

readOptions - options to pass to the Spark read operation

Returns:

Spark Dataset containing the training dataset data

Throws:

FeatureStoreException - if the training dataset has splits and the split was not specified

IOException - IOException

show

public void show(int numRows)
          throws FeatureStoreException,
IOException

Show numRows from the training dataset (across all splits).

Parameters:

numRows - number of rows to display

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

computeStatistics

public Statistics computeStatistics()
                             throws FeatureStoreException,
IOException

Recompute the statistics for the entire training dataset and save them to the feature store.

Returns:

statistics object of computed statistics

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException