com.logicalclocks.hsfs.spark

Class ExternalFeatureGroup

java.lang.Object

com.logicalclocks.hsfs.FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

com.logicalclocks.hsfs.spark.ExternalFeatureGroup

public class ExternalFeatureGroup
extends FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Field Summary

Fields inherited from class com.logicalclocks.hsfs.FeatureGroupBase

created, creator, dataSource, deltaStreamerJobConf, deprecated, description, eventTime, expectationsNames, featureGroupEngineBase, features, featureStore, hudiPrecombineKey, id, location, LOGGER, name, notificationTopicName, onlineConfig, onlineEnabled, onlineIngestionApi, onlineTopicName, partitionKeys, primaryKeys, statisticColumns, statisticsConfig, storageConnector, subject, timeTravelFormat, topicName, type, utils, version

Constructor Summary

Constructors

Constructor and Description
ExternalFeatureGroup()
ExternalFeatureGroup(FeatureStore featureStore, int id)
ExternalFeatureGroup(FeatureStore featureStore, @NonNull String name, Integer version, String query, ExternalDataFormat dataFormat, String path, Map<String,String> options, @NonNull StorageConnector storageConnector, String description, List<String> primaryKeys, List<Feature> features, StatisticsConfig statisticsConfig, String eventTime, boolean onlineEnabled, String onlineTopicName, String topicName, String notificationTopicName, OnlineConfig onlineConfig)

Method Summary

Modifier and Type	Method and Description
void	appendFeatures(Feature features) Append a single feature to the schema of the feature group.
void	appendFeatures(List<Feature> features) Append features to the schema of the feature group.
QueryBase	asOf(String wallclockTime)
QueryBase	asOf(String wallclockTime, String excludeUntil)
void	commitDeleteRecord(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData)
void	commitDeleteRecord(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, Map<String,String> writeOptions)
Map<Long,Map<String,String>>	commitDetails()
Map<Long,Map<String,String>>	commitDetails(Integer limit)
Map<Long,Map<String,String>>	commitDetails(String wallclockTime)
Map<Long,Map<String,String>>	commitDetails(String wallclockTime, Integer limit)
Statistics	computeStatistics() Recompute the statistics for the feature group and save them to the feature store.
Statistics	computeStatistics(String wallclockTime)
Statistics	getStatistics() Get the last statistics commit for the feature group.
void	insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData) Incrementally insert data to the online storage of an external feature group.
void	insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, boolean overwrite)
void	insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, boolean overwrite, Map<String,String> writeOptions)
void	insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, boolean overwrite, Map<String,String> writeOptions, JobConfiguration jobConfiguration)
void	insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, HudiOperationType operation)
void	insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, JobConfiguration jobConfiguration)
void	insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, Map<String,String> writeOptions) Incrementally insert data to the online storage of an external feature group.
void	insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, Storage storage)
void	insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, Storage storage, boolean overwrite)
void	insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, Storage storage, boolean overwrite, HudiOperationType operation, Map<String,String> writeOptions)
Object	insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData)
Object	insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, Map<String,String> writeOptions)
Object	insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, String queryName)
Object	insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, String queryName, Map<String,String> writeOptions)
Object	insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, String queryName, String outputMode)
Object	insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, String queryName, String outputMode, boolean awaitTermination, Long timeout)
Object	insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, String queryName, String outputMode, boolean awaitTermination, Long timeout, String checkpointLocation)
Object	insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, String queryName, String outputMode, boolean awaitTermination, Long timeout, String checkpointLocation, Map<String,String> writeOptions)
Object	insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, String queryName, String outputMode, boolean awaitTermination, Long timeout, String checkpointLocation, Map<String,String> writeOptions, JobConfiguration jobConfiguration)
Object	insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, String queryName, String outputMode, boolean awaitTermination, String checkpointLocation)
Object	insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData, String queryName, String outputMode, String checkpointLocation)
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read()
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(boolean online)
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(boolean online, Map<String,String> readOptions)
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(Map<String,String> readOptions)
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(String wallclockTime)
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	read(String wallclockTime, Map<String,String> readOptions)
void	save()
Query	select(List<String> features)
Query	selectAll()
Query	selectExcept(List<String> features)
Query	selectExceptFeatures(List<Feature> features)
Query	selectFeatures(List<Feature> features)
void	show(int numRows)
void	show(int numRows, boolean online)
void	updateFeatures(Feature feature) Update the metadata of multiple features.
void	updateFeatures(List<Feature> features) Update the metadata of multiple features.

Methods inherited from class com.logicalclocks.hsfs.FeatureGroupBase

addTag, checkDeprecated, delete, deleteTag, getAvroSchema, getComplexFeatures, getDeserializedAvroSchema, getDeserializedEncodedAvroSchema, getEncodedAvroSchema, getFeature, getFeatureAvroSchema, getLatestOnlineIngestion, getOnlineIngestion, getPrimaryKeys, getSubject, getTag, getTags, setDeprecated, unloadSubject, updateDeprecated, updateDeprecated, updateDescription, updateFeatureDescription, updateNotificationTopicName, updateStatisticsConfig

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ExternalFeatureGroup

public ExternalFeatureGroup(FeatureStore featureStore,
                            @NonNull
                            @NonNull String name,
                            Integer version,
                            String query,
                            ExternalDataFormat dataFormat,
                            String path,
                            Map<String,String> options,
                            @NonNull
                            @NonNull StorageConnector storageConnector,
                            String description,
                            List<String> primaryKeys,
                            List<Feature> features,
                            StatisticsConfig statisticsConfig,
                            String eventTime,
                            boolean onlineEnabled,
                            String onlineTopicName,
                            String topicName,
                            String notificationTopicName,
                            OnlineConfig onlineConfig)

ExternalFeatureGroup

public ExternalFeatureGroup()

ExternalFeatureGroup

public ExternalFeatureGroup(FeatureStore featureStore,
                            int id)

Method Detail

save

public void save()
          throws FeatureStoreException,
                 IOException

Throws:

FeatureStoreException

IOException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read()
                                                            throws FeatureStoreException,
                                                                   IOException

Specified by:

read in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(boolean online)
                                                            throws FeatureStoreException,
                                                                   IOException

Specified by:

read in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(Map<String,String> readOptions)
                                                            throws FeatureStoreException,
                                                                   IOException

Specified by:

read in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(boolean online,
                                                                   Map<String,String> readOptions)
                                                            throws FeatureStoreException,
                                                                   IOException

Specified by:

read in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(String wallclockTime)
                                                            throws FeatureStoreException,
                                                                   IOException,
                                                                   ParseException

Specified by:

read in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

ParseException

read

public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> read(String wallclockTime,
                                                                   Map<String,String> readOptions)
                                                            throws FeatureStoreException,
                                                                   IOException,
                                                                   ParseException

Specified by:

read in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

ParseException

asOf

public QueryBase asOf(String wallclockTime)
               throws FeatureStoreException,
                      ParseException

Specified by:

asOf in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

ParseException

asOf

public QueryBase asOf(String wallclockTime,
                      String excludeUntil)
               throws FeatureStoreException,
                      ParseException

Specified by:

asOf in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

ParseException

show

public void show(int numRows)
          throws FeatureStoreException,
                 IOException

Specified by:

show in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

show

public void show(int numRows,
                 boolean online)
          throws FeatureStoreException,
                 IOException

Specified by:

show in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

insert

public void insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                   Storage storage)
            throws IOException,
                   FeatureStoreException,
                   ParseException

Specified by:

insert in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

IOException

FeatureStoreException

ParseException

insert

public void insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                   boolean overwrite)
            throws IOException,
                   FeatureStoreException,
                   ParseException

Specified by:

insert in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

IOException

FeatureStoreException

ParseException

insert

public void insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                   Storage storage,
                   boolean overwrite)
            throws IOException,
                   FeatureStoreException,
                   ParseException

Specified by:

insert in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

IOException

FeatureStoreException

ParseException

insert

public void insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                   boolean overwrite,
                   Map<String,String> writeOptions)
            throws FeatureStoreException,
                   IOException,
                   ParseException

Specified by:

insert in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

ParseException

insert

public void insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                   HudiOperationType operation)
            throws FeatureStoreException,
                   IOException,
                   ParseException

Specified by:

insert in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

ParseException

insert

public void insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                   Storage storage,
                   boolean overwrite,
                   HudiOperationType operation,
                   Map<String,String> writeOptions)
            throws FeatureStoreException,
                   IOException,
                   ParseException

Specified by:

insert in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

ParseException

insert

public void insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                   JobConfiguration jobConfiguration)
            throws FeatureStoreException,
                   IOException,
                   ParseException

Specified by:

insert in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

ParseException

insert

public void insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                   boolean overwrite,
                   Map<String,String> writeOptions,
                   JobConfiguration jobConfiguration)
            throws FeatureStoreException,
                   IOException,
                   ParseException

Specified by:

insert in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

ParseException

insert

public void insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData)
            throws FeatureStoreException,
                   IOException

Incrementally insert data to the online storage of an external feature group. The feature group has to be online enabled to perform this operation. The `features` dataframe can be a Spark DataFrame or RDD. If statistics are enabled, statistics are recomputed for the entire feature group. If the feature group doesn't exist, the insert method will create the necessary metadata the first time it is invoked and write the specified `features` dataframe as feature group to the online feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature group handle
        ExternalFeatureGroup fg = fs.getExternalFeatureGroup("electricity_prices", 1);
        // insert data
        fg.insert(featureData, writeOptions);
 
 

Specified by:

insert in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Parameters:

featureData - Spark DataFrame, RDD. Features to be saved.

Throws:

IOException - Generic IO exception.

FeatureStoreException - If client is not connected to Hopsworks; cannot run read query on storage and/or can't reconcile schema.

insert

public void insert(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                   Map<String,String> writeOptions)
            throws FeatureStoreException,
                   IOException,
                   ParseException

Incrementally insert data to the online storage of an external feature group. The feature group has to be online enabled to perform this operation. The `features` dataframe can be a Spark DataFrame or RDD. If statistics are enabled, statistics are recomputed for the entire feature group. If the feature group doesn't exist, the insert method will create the necessary metadata the first time it is invoked and write the specified `features` dataframe as feature group to the online feature store.

 
        // get feature store handle
        FeatureStore fs = HopsworksConnection.builder().build().getFeatureStore();
        // get feature group handle
        ExternalFeatureGroup fg = fs.getExternalFeatureGroup("electricity_prices", 1);
        // Define additional write options (for example for Spark)
        Map<String, String> writeOptions = = new HashMap<String, String>();
        // insert data
        fg.insert(featureData, writeOptions);
 
 

Specified by:

insert in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Parameters:

featureData - Spark DataFrame, RDD. Features to be saved.

writeOptions - Additional write options as key-value pairs.

Throws:

IOException - Generic IO exception.

FeatureStoreException - If client is not connected to Hopsworks; cannot run read query on storage and/or can't reconcile schema.

ParseException

commitDeleteRecord

public void commitDeleteRecord(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData)
                        throws FeatureStoreException,
                               IOException,
                               ParseException

Specified by:

commitDeleteRecord in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

ParseException

commitDeleteRecord

public void commitDeleteRecord(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                               Map<String,String> writeOptions)
                        throws FeatureStoreException,
                               IOException,
                               ParseException

Specified by:

commitDeleteRecord in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

ParseException

commitDetails

public Map<Long,Map<String,String>> commitDetails()
                                           throws IOException,
                                                  FeatureStoreException,
                                                  ParseException

Specified by:

commitDetails in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

IOException

FeatureStoreException

ParseException

commitDetails

public Map<Long,Map<String,String>> commitDetails(Integer limit)
                                           throws IOException,
                                                  FeatureStoreException,
                                                  ParseException

Specified by:

commitDetails in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

IOException

FeatureStoreException

ParseException

commitDetails

public Map<Long,Map<String,String>> commitDetails(String wallclockTime)
                                           throws IOException,
                                                  FeatureStoreException,
                                                  ParseException

Specified by:

commitDetails in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

IOException

FeatureStoreException

ParseException

commitDetails

public Map<Long,Map<String,String>> commitDetails(String wallclockTime,
                                                  Integer limit)
                                           throws IOException,
                                                  FeatureStoreException,
                                                  ParseException

Specified by:

commitDetails in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

IOException

FeatureStoreException

ParseException

selectFeatures

public Query selectFeatures(List<Feature> features)

Specified by:

selectFeatures in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

select

public Query select(List<String> features)

Specified by:

select in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

selectAll

public Query selectAll()

Specified by:

selectAll in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

selectExceptFeatures

public Query selectExceptFeatures(List<Feature> features)

Specified by:

selectExceptFeatures in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

selectExcept

public Query selectExcept(List<String> features)

Specified by:

selectExcept in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

insertStream

public Object insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData)
                    throws Exception

Specified by:

insertStream in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

Exception

insertStream

public Object insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                           String queryName)
                    throws Exception

Specified by:

insertStream in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

Exception

insertStream

public Object insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                           Map<String,String> writeOptions)
                    throws Exception

Specified by:

insertStream in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

Exception

insertStream

public Object insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                           String queryName,
                           Map<String,String> writeOptions)
                    throws Exception

Specified by:

insertStream in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

Exception

insertStream

public Object insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                           String queryName,
                           String outputMode)
                    throws Exception

Specified by:

insertStream in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

Exception

insertStream

public Object insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                           String queryName,
                           String outputMode,
                           String checkpointLocation)
                    throws Exception

Specified by:

insertStream in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

Exception

insertStream

public Object insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                           String queryName,
                           String outputMode,
                           boolean awaitTermination,
                           Long timeout)
                    throws Exception

Specified by:

insertStream in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

Exception

insertStream

public Object insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                           String queryName,
                           String outputMode,
                           boolean awaitTermination,
                           Long timeout,
                           String checkpointLocation)
                    throws Exception

Specified by:

insertStream in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

Exception

insertStream

public Object insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                           String queryName,
                           String outputMode,
                           boolean awaitTermination,
                           Long timeout,
                           String checkpointLocation,
                           Map<String,String> writeOptions)
                    throws Exception

Specified by:

insertStream in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

Exception

insertStream

public Object insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                           String queryName,
                           String outputMode,
                           boolean awaitTermination,
                           String checkpointLocation)
                    throws Exception

Specified by:

insertStream in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

Exception

insertStream

public Object insertStream(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> featureData,
                           String queryName,
                           String outputMode,
                           boolean awaitTermination,
                           Long timeout,
                           String checkpointLocation,
                           Map<String,String> writeOptions,
                           JobConfiguration jobConfiguration)
                    throws Exception

Specified by:

insertStream in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

Exception

updateFeatures

public void updateFeatures(List<Feature> features)
                    throws FeatureStoreException,
                           IOException,
                           ParseException

Description copied from class: FeatureGroupBase

Update the metadata of multiple features. Currently only feature description updates are supported.

Specified by:

updateFeatures in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Parameters:

features - List of Feature metadata objects

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

ParseException - ParseException

updateFeatures

public void updateFeatures(Feature feature)
                    throws FeatureStoreException,
                           IOException,
                           ParseException

Description copied from class: FeatureGroupBase

Update the metadata of multiple features. Currently only feature description updates are supported.

Specified by:

updateFeatures in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Parameters:

feature - Feature metadata object

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

ParseException - ParseException

appendFeatures

public void appendFeatures(List<Feature> features)
                    throws FeatureStoreException,
                           IOException,
                           ParseException

Description copied from class: FeatureGroupBase

Append features to the schema of the feature group. It is only possible to append features to a feature group. Removing features is considered a breaking change.

Specified by:

appendFeatures in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Parameters:

features - list of Feature metadata objects

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

ParseException - ParseException

appendFeatures

public void appendFeatures(Feature features)
                    throws FeatureStoreException,
                           IOException,
                           ParseException

Description copied from class: FeatureGroupBase

Append a single feature to the schema of the feature group. It is only possible to append features to a feature group. Removing features is considered a breaking change.

Specified by:

appendFeatures in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Parameters:

features - List of Feature metadata objects

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

ParseException - ParseException

computeStatistics

public Statistics computeStatistics()
                             throws FeatureStoreException,
                                    IOException

Description copied from class: FeatureGroupBase

Recompute the statistics for the feature group and save them to the feature store.

Specified by:

computeStatistics in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Returns:

statistics object of computed statistics

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException

computeStatistics

public Statistics computeStatistics(String wallclockTime)
                             throws FeatureStoreException,
                                    IOException,
                                    ParseException

Specified by:

computeStatistics in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Throws:

FeatureStoreException

IOException

ParseException

getStatistics

public Statistics getStatistics()
                         throws FeatureStoreException,
                                IOException

Description copied from class: FeatureGroupBase

Get the last statistics commit for the feature group.

Specified by:

getStatistics in class FeatureGroupBase<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

Returns:

statistics object of latest commit

Throws:

FeatureStoreException - FeatureStoreException

IOException - IOException