com.logicalclocks.hsfs.spark.engine

Class TrainingDatasetEngine

java.lang.Object

com.logicalclocks.hsfs.spark.engine.TrainingDatasetEngine

public class TrainingDatasetEngine
extends Object

Constructor Summary

Constructors

Constructor and Description
TrainingDatasetEngine()

Method Summary

Modifier and Type	Method and Description
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(TrainingDataset trainingDataset, String split, Map<String,String> providedOptions)
TrainingDataset	save(TrainingDataset trainingDataset, Query query, Map<String,String> userWriteOptions, List<String> labels) Make a REST call to Hopsworks to create the metadata and write the data on the File System.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TrainingDatasetEngine

public TrainingDatasetEngine()

Method Detail

save

public TrainingDataset save(TrainingDataset trainingDataset,
                            Query query,
                            Map<String,String> userWriteOptions,
                            List<String> labels)
                     throws FeatureStoreException,
                            IOException

Make a REST call to Hopsworks to create the metadata and write the data on the File System.

Parameters:

trainingDataset - Training Dataset metadata object

query - Query Object

userWriteOptions - Additional write options as key-value pairs, defaults to empty Map

Returns:

Training Dataset metadata object

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(TrainingDataset trainingDataset,
                                                                   String split,
                                                                   Map<String,String> providedOptions)
                                                            throws FeatureStoreException,
                                                                   IOException

Throws:

FeatureStoreException

IOException