Skip to content

Training Dataset#

[source]

TrainingDataset#

hsfs.training_dataset.TrainingDataset(
    name,
    version,
    data_format,
    location,
    featurestore_id,
    description=None,
    storage_connector=None,
    splits=None,
    seed=None,
    created=None,
    creator=None,
    features=None,
    statistics_config=None,
    featurestore_name=None,
    id=None,
    jobs=None,
    inode_id=None,
    training_dataset_type=None,
    from_query=None,
    querydto=None,
    label=None,
)

Creation#

[source]

create_training_dataset#

FeatureStore.create_training_dataset(
    name,
    version=None,
    description="",
    data_format="tfrecords",
    storage_connector=None,
    splits={},
    location="",
    seed=None,
    statistics_config=None,
    label=[],
)

Create a training dataset metadata object.

Lazy

This method is lazy and does not persist any metadata or feature data in the feature store on its own. To materialize the training dataset and save feature data along the metadata in the feature store, call the save() method with a DataFrame or Query.

Data Formats

The feature store currently supports the following data formats for training datasets:

  1. tfrecord
  2. csv
  3. tsv
  4. parquet
  5. avro
  6. orc

Currently not supported petastorm, hdf5 and npy file formats.

Arguments

  • name str: Name of the training dataset to create.
  • version Optional[int]: Version of the training dataset to retrieve, defaults to None and will create the training dataset with incremented version from the last version in the feature store.
  • description Optional[str]: A string describing the contents of the training dataset to improve discoverability for Data Scientists, defaults to empty string "".
  • data_format Optional[str]: The data format used to save the training dataset, defaults to "tfrecords"-format.
  • storage_connector Optional[hsfs.StorageConnector]: Storage connector defining the sink location for the training dataset, defaults to None, and materializes training dataset on HopsFS.
  • splits Optional[Dict[str, float]]: A dictionary defining training dataset splits to be created. Keys in the dictionary define the name of the split as str, values represent percentage of samples in the split as float. Currently, only random splits are supported. Defaults to empty dict{}, creating only a single training dataset without splits.
  • location Optional[str]: Path to complement the sink storage connector with, e.g if the storage connector points to an S3 bucket, this path can be used to define a sub-directory inside the bucket to place the training dataset. Defaults to "", saving the training dataset at the root defined by the storage connector.
  • seed Optional[int]: Optionally, define a seed to create the random splits with, in order to guarantee reproducability, defaults to None.
  • statistics_config Optional[Union[hsfs.StatisticsConfig, bool, dict]]: A configuration object, or a dictionary with keys "enabled" to generally enable descriptive statistics computation for this feature group, "correlations" to turn on feature correlation computation and "histograms" to compute feature value frequencies. The values should be booleans indicating the setting. To fully turn off statistics computation pass statistics_config=False. Defaults to None and will compute only descriptive statistics.
  • label Optional[List[str]]: A list of feature names constituting the prediction label/feature of the training dataset. When replaying a Query during model inference, the label features can be omitted from the feature vector retrieval. Defaults to [], no label.

Returns:

TrainingDataset: The training dataset metadata object.

Retrieval#

[source]

get_training_dataset#

FeatureStore.get_training_dataset(name, version=None)

Get a training dataset entity from the feature store.

Getting a training dataset from the Feature Store means getting its metadata handle so you can subsequently read the data into a Spark or Pandas DataFrame.

Arguments

  • name str: Name of the training dataset to get.
  • version Optional[int]: Version of the training dataset to retrieve, defaults to None and will return the version=1.

Returns

TrainingDataset: The training dataset metadata object.

Raises

  • RestAPIError: If unable to retrieve feature group from the feature store.

Properties#

[source]

data_format#

File format of the training dataset.

[source]

description#

[source]

feature_store_id#

[source]

id#

Training dataset id.

[source]

label#

The label/prediction feature of the training dataset.

Can be a composite of multiple features.

[source]

location#

Path to the training dataset location.

[source]

name#

Name of the training dataset.

[source]

query#

Query to generate this training dataset from online feature store.

[source]

schema#

Training dataset schema.

[source]

seed#

Seed.

[source]

splits#

Training dataset splits. train, test or eval and corresponding percentages.

[source]

statistics#

Get the latest computed statistics for the training dataset.

[source]

statistics_config#

Statistics configuration object defining the settings for statistics computation of the training dataset.

[source]

storage_connector#

Storage connector.

[source]

version#

Version number of the training dataset.

[source]

write_options#

User provided options to write training dataset.

Methods#

[source]

add_tag#

TrainingDataset.add_tag(name, value=None)

Attach a name/value tag to a training dataset.

A tag can consist of a name only or a name/value pair. Tag names are unique identifiers.

Arguments

  • name str: Name of the tag to be added.
  • value Optional[str]: Value of the tag to be added, defaults to None.

[source]

compute_statistics#

TrainingDataset.compute_statistics()

Recompute the statistics for the training dataset and save them to the feature store.

[source]

delete_tag#

TrainingDataset.delete_tag(name)

Delete a tag from a training dataset.

Tag names are unique identifiers.

Arguments

  • name str: Name of the tag to be removed.

[source]

from_response_json#

TrainingDataset.from_response_json(json_dict)

[source]

get_query#

TrainingDataset.get_query(online=True, with_label=False)

Returns the query used to generate this training dataset

Arguments

  • online bool: boolean, optional. Return the query for the online storage, else for offline storage, defaults to True - for online storage.
  • with_label bool: Indicator whether the query should contain features which were marked as prediction label/feature when the training dataset was created, defaults to False.

Returns

str. Query string for the chosen storage used to generate this training dataset.

[source]

get_statistics#

TrainingDataset.get_statistics(commit_time=None)

Returns the statistics for this training dataset at a specific time.

If commit_time is None, the most recent statistics are returned.

Arguments

  • commit_time Optional[str]: Commit time in the format YYYYMMDDhhmmss, defaults to None.

Returns

Statistics. Object with statistics information.

[source]

get_tag#

TrainingDataset.get_tag(name=None)

Get the tags of a training dataset.

Tag names are unique identifiers. Returns all tags if no tag name is specified.

Arguments

  • name: Name of the tag to get, defaults to None.

Returns

List[Tag]. List of tags as name/value pairs.

[source]

insert#

TrainingDataset.insert(features, overwrite, write_options={})

Insert additional feature data into the training dataset.

This method appends data to the training dataset either from a Feature Store Query, a Spark or Pandas DataFrame, a Spark RDD, two-dimensional Python lists or Numpy ndarrays. The schemas must match for this operation.

This can also be used to overwrite all data in an existing training dataset.

Arguments

  • features Union[hsfs.constructor.query.Query, pandas.DataFrame, pyspark.sql.DataFrame, pyspark.RDD, numpy.ndarray, List[list]]: Feature data to be materialized.
  • overwrite bool: Whether to overwrite the entire data in the training dataset.
  • write_options Optional[Dict[Any, Any]]: Additional write options as key/value pairs. Defaults to {}.

Returns

TrainingDataset: The updated training dataset metadata object, the previous TrainingDataset object on which you call save is also updated.

Raises

  • RestAPIError: Unable to create training dataset metadata.

[source]

json#

TrainingDataset.json()

[source]

read#

TrainingDataset.read(split=None, read_options={})

Read the training dataset into a dataframe.

It is also possible to read only a specific split.

Arguments

  • split: Name of the split to read, defaults to None, reading the entire training dataset.
  • read_options: Additional read options as key/value pairs, defaults to {}.

Returns

DataFrame: The spark dataframe containing the feature data of the training dataset.

[source]

save#

TrainingDataset.save(features, write_options={})

Materialize the training dataset to storage.

This method materializes the training dataset either from a Feature Store Query, a Spark or Pandas DataFrame, a Spark RDD, two-dimensional Python lists or Numpy ndarrays.

Arguments

  • features Union[hsfs.constructor.query.Query, pandas.DataFrame, pyspark.sql.DataFrame, pyspark.RDD, numpy.ndarray, List[list]]: Feature data to be materialized.
  • write_options Optional[Dict[Any, Any]]: Additional write options as key/value pairs. Defaults to {}.

Returns

TrainingDataset: The updated training dataset metadata object, the previous TrainingDataset object on which you call save is also updated.

Raises

  • RestAPIError: Unable to create training dataset metadata.

[source]

show#

TrainingDataset.show(n, split=None)

Show the first n rows of the training dataset.

You can specify a split from which to retrieve the rows.

Arguments

  • n int: Number of rows to show.
  • split Optional[str]: Name of the split to show, defaults to None, showing the first rows when taking all splits together.

[source]

tf_data#

TrainingDataset.tf_data(
    target_name,
    split=None,
    feature_names=None,
    var_len_features=[],
    is_training=True,
    cycle_length=2,
)

Returns an object with utility methods to read training dataset as tf.data.Dataset object and handle it for further processing.

Arguments

  • target_name str: Name of the target variable.
  • split Optional[str]: Name of training dataset split. For example, "train", "test" or "val", defaults to None, returning the full training dataset.
  • feature_names Optional[list]: Names of training variables, defaults to None.
  • var_len_features Optional[list]: Feature names that have variable length and need to be returned as tf.io.VarLenFeature, defaults to [].
  • is_training Optional[bool]: Whether it is for training, testing or validation. Defaults to True.
  • cycle_length Optional[int]: Number of files to be read and deserialized in parallel, defaults to 2.

Returns

TFDataEngine. An object with utility methods to generate and handle tf.data.Dataset object.

[source]

to_dict#

TrainingDataset.to_dict()

[source]

update_from_response_json#

TrainingDataset.update_from_response_json(json_dict)

[source]

update_statistics_config#

TrainingDataset.update_statistics_config()

Update the statistics configuration of the training dataset.

Change the statistics_config object and persist the changes by calling this method.

Returns

TrainingDataset. The updated metadata object of the training dataset.

Raises

RestAPIError.