Skip to content

Transformation Functions#

HSFS provides functionality to attach transformation functions to training datasets.

User defined, custom transformation functions need to be registered in the Feature Store to make them accessible for training dataset creation. To register them in the Feature Store they either have to be part of the library installed in Hopsworks or attached when starting a Jupyter notebook or Hopsworks job.

Pyspark decorators

Don't decorate transformation functions with Pyspark @udf or @pandas_udf, as well as don't use any Pyspark dependencies. HSFS will decorate transformation function only if it is used inside Pyspark application.

Hopsworks also ships Built-in transformation functions such as min_max_scaler, standard_scaler, robust_scaler and label_encoder. They can be registered by callingregister_builtin_transformation_functions method on the feature store handle.

Examples#

Let's assume that we have already installed Python library transformation_fn_template containing transformation function plus_one.

Register transformation function plus_one in the Hopsworks feature store.

from custom_functions import transformations
plus_one_meta = fs.create_transformation_function(
            transformation_function=transformations.plus_one,
            output_type=int,
            version=1)
plus_one_meta.save()

To retrieve all transformation functions from the feature store use get_transformation_functions which will return list of TransformationFunction objects. A specific transformation function can be retrieved by get_transformation_function method where the user can provide a name and a version of the transformation function. If only function name is provided then it will default to version 1.

Retrieving transformation functions from the feature store

# get all transformation functions
fs.get_transformation_functions()

# get transformation function by name. This will default to version 1
fs.get_transformation_function(name="plus_one")

# get built-in transformation function min max scaler
fs.get_transformation_function(name="min_max_scaler")

# get transformation function by name and version.
fs.get_transformation_function(name="plus_one", version=2)

To attach transformation function to training dataset provide transformation functions as dict, where key is feature name and value is online transformation function name. Also training dataset must be created from the Query object. Once attached transformation function will be applied on whenever save, insert and get_serving_vector methods are called on training dataset object.

Attaching transformation functions to the training dataset

plus_one_meta = fs.get_transformation_function(name="plus_one", version=1)
fs.create_training_dataset(name="td_demo",
                           description="Dataset to train the demo model",
                           data_format="csv",
                           transformation_functions={"feature_name":plus_one_meta}
                           statistics_config=None,
                           version=1)
td.save(join_query)

Built-in transformation functions are attached in the same way. The only difference is that it will compute the necessary statistics for the specific function in the background. For example min and max values for min_max_scaler; mean and standard deviation for standard_scaler etc.

Attaching built-in transformation functions to the training dataset

min_max_scaler = fs.get_transformation_function(name="min_max_scaler")
standard_scaler = fs.get_transformation_function(name="standard_scaler")
robust_scaler = fs.get_transformation_function(name="robust_scaler")
label_encoder = fs.get_transformation_function(name="label_encoder")
fs.create_training_dataset(name="td_demo",
                           description="Dataset to train the demo model",
                           data_format="csv",
                           transformation_functions={"feature_name":min_max_scaler,
                                                     "feature_name":standard_scaler,
                                                     "feature_name":robust_scaler,
                                                     "feature_name":label_encoder},
                           statistics_config=None,
                           version=1)
td.save(join_query)

Scala support

Creating and attaching Transformation functions to training datasets are not supported for hsfs scala client. If training dataset with transformation function was created using python client and later insert or getServingVector methods are called on this training dataset from scala client hsfs will throw an exception.

Transformation Function#

[source]

TransformationFunction#

hsfs.transformation_function.TransformationFunction(
    featurestore_id,
    transformation_fn=None,
    version=None,
    name=None,
    source_code_content=None,
    builtin_source_code=None,
    output_type=None,
    id=None,
    type=None,
    items=None,
    count=None,
    href=None,
)

Properties#

[source]

id#

Training dataset id.


[source]

name#


[source]

output_type#


[source]

source_code_content#


[source]

transformation_fn#


[source]

transformer_code#


[source]

version#


Methods#

[source]

delete#

TransformationFunction.delete()

Delete transformation function from backend.


[source]

save#

TransformationFunction.save()

Persist transformation function in backend.


Creation#

[source]

create_transformation_function#

FeatureStore.create_transformation_function(transformation_function, output_type, version=None)

Create a transformation function metadata object.

Lazy

This method is lazy and does not persist the transformation function in the feature store on its own. To materialize the transformation function and save call the save() method of the transformation function metadata object.

Arguments

  • transformation_function callable: callable object.
  • output_type Union[str, bytes, int, numpy.int8, numpy.int16, numpy.int32, numpy.int64, float, numpy.float64, datetime.datetime, numpy.datetime64, datetime.date, bool]: python or numpy output type that will be inferred as pyspark.sql.types type.

Returns:

TransformationFunction: The TransformationFunction metadata object.


Retrieval#

[source]

get_transformation_function#

FeatureStore.get_transformation_function(name, version=None)

Get transformation function metadata object.

Arguments

  • name str: name of transformation function.
  • version Optional[int]: version of transformation function. Optional, if not provided all functions that match to provided name will be retrieved .

Returns:

TransformationFunction: The TransformationFunction metadata object.


[source]

get_transformation_functions#

FeatureStore.get_transformation_functions()

Get all transformation functions metadata objects.

Returns:

List[TransformationFunction]. List of transformation function instances.