Skip to content

HopsworksUDF#

[source]

HopsworksUdf#

hsfs.hopsworks_udf.HopsworksUdf(
    func,
    return_types,
    name=None,
    transformation_features=None,
    transformation_function_argument_names=None,
    dropped_argument_names=None,
    dropped_feature_names=None,
    feature_name_prefix=None,
)

Meta data for user defined functions.

Stores meta data required to execute the user defined function in both spark and python engine. The class generates uses the metadata to dynamically generate user defined functions based on the engine it is executed in.

Arguments

  • func : Union[Callable, str]. The transformation function object or the source code of the transformation function.
  • return_types : Union[List[type], type, List[str], str]. A python type or a list of python types that denotes the data types of the columns output from the transformation functions.
  • name : Optional[str]. Name of the transformation function.
  • transformation_features : Optional[List[TransformationFeature]]. A list of objects of TransformationFeature that maps the feature used for transformation to their corresponding statistics argument names if any
  • transformation_function_argument_names : Optional[List[TransformationFeature]]. The argument names of the transformation function.
  • dropped_argument_names : Optional[List[str]]. The arguments to be dropped from the finial DataFrame after the transformation functions are applied.
  • dropped_feature_names : Optional[List[str]]. The feature name corresponding to the arguments names that are dropped
  • feature_name_prefix str | None: Optional[str] = None. Prefixes if any used in the feature view.

Properties#

[source]

dropped_features#

List of features that will be dropped after the UDF is applied.

[source]

function_name#

Get the function name of the UDF

[source]

output_column_names#

Output columns names of the transformation function

[source]

return_types#

Get the output types of the UDF

[source]

statistics_features#

List of feature names that require statistics

[source]

statistics_required#

Get if statistics for any feature is required by the UDF

[source]

transformation_features#

List of feature names to be used in the User Defined Function.

[source]

transformation_statistics#

Feature statistics required for the defined UDF

TransformationFeature#

[source]

TransformationFeature#

hsfs.hopsworks_udf.TransformationFeature(feature_name, statistic_argument_name)

Mapping of feature names to their corresponding statistics argument names in the code.

The statistic_argument_name for a feature name would be None if the feature does not need statistics.

Arguments

  • feature_name : str. Name of the feature.
  • statistic_argument_name : str. Name of the statistics argument in the code for the feature specified in the feature name.